[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:31:38 dist-test-slave-04bb krb5kdc[9422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:31:38 dist-test-slave-04bb krb5kdc[9422](info): set up 2 sockets
Aug 10 23:31:38 dist-test-slave-04bb krb5kdc[9422](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 23:31:40.226203 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.551s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:31:40 dist-test-slave-04bb krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:31:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:31:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:31:40.431314 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:40155
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:40.800755 9440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:40.801189 9440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:40.801373 9440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:40.809275 9440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:40.809418 9440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:40.809499 9440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:40.809578 9440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:40.809635 9440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:40.821244 9440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:40155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:40155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:40.823164 9440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:40.825474 9440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:31:40.844298 9440 server_base.cc:1034] running on GCE node
W20260810 23:31:40.844278 9452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:40.844292 9451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:40.844384 9457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:40.847657 9440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:40.849856 9440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:40.851348 9440 hybrid_clock.cc:648] HybridClock initialized: now 1786404700851329 us; error 95 us; skew 500 ppm
I20260810 23:31:40.851881 9440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:40 dist-test-slave-04bb krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404700, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:40.855787 9440 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:31:40.858336 9440 webserver.cc:460] Webserver started at http://127.9.48.254:34983/ using document root <none> and password file <none>
I20260810 23:31:40.859488 9440 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:40.859639 9440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:40.860092 9440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:40.864603 9440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "833a5c64818042678a463cba54817429"
format_stamp: "Formatted at 2026-08-10 23:31:40 on dist-test-slave-04bb"
server_key: "cbf05157721f6c345c9e4633ad46f6a2"
server_key_iv: "7e15dc9bcc7e8410ccb1e5172b93e8e2"
server_key_version: "encryptionkey@0"
I20260810 23:31:40.865468 9440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "833a5c64818042678a463cba54817429"
format_stamp: "Formatted at 2026-08-10 23:31:40 on dist-test-slave-04bb"
server_key: "cbf05157721f6c345c9e4633ad46f6a2"
server_key_iv: "7e15dc9bcc7e8410ccb1e5172b93e8e2"
server_key_version: "encryptionkey@0"
I20260810 23:31:40.874054 9440 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260810 23:31:40.879343 9466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:40.881242 9440 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 23:31:40.881460 9440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "833a5c64818042678a463cba54817429"
format_stamp: "Formatted at 2026-08-10 23:31:40 on dist-test-slave-04bb"
server_key: "cbf05157721f6c345c9e4633ad46f6a2"
server_key_iv: "7e15dc9bcc7e8410ccb1e5172b93e8e2"
server_key_version: "encryptionkey@0"
I20260810 23:31:40.881685 9440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:40.908452 9440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:40.913949 9440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:40.914322 9440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:40.938382 9440 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:40155
I20260810 23:31:40.938470 9550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:40155 every 8 connection(s)
I20260810 23:31:40.940723 9440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:31:40.943060 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 9440
I20260810 23:31:40.943538 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:31:40.947165 9411 external_mini_cluster.cc:1428] Setting key e1da7b7d5835461e76b46c19876cdc88
I20260810 23:31:40.949337 9552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:40.971776 9552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429: Bootstrap starting.
I20260810 23:31:40.978686 9552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:40.980543 9552 log.cc:824] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:40.985149 9552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429: No bootstrap required, opened a new log
Aug 10 23:31:40 dist-test-slave-04bb krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:40.997807 9552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833a5c64818042678a463cba54817429" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40155 } }
I20260810 23:31:40.998441 9552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:40.998662 9552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833a5c64818042678a463cba54817429, State: Initialized, Role: FOLLOWER
I20260810 23:31:40.999962 9552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833a5c64818042678a463cba54817429" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40155 } }
I20260810 23:31:41.000336 9552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:31:41.000674 9552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:31:41.000914 9552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:41.002647 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:40.956171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48538 (local address 127.9.48.254:40155)
0810 23:31:40.958566 (+ 2395us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:40.958577 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:40.958634 (+ 57us) server_negotiation.cc:407] Connection header received
0810 23:31:40.971630 (+ 12996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:40.971664 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:40.972230 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:40.972724 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:40.977927 (+ 5203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:40.979489 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:40.982258 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:40.983032 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:40.983242 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:40.991525 (+ 8283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:40.991557 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:40.991588 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:40.991643 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:40.996198 (+ 4555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:40.997197 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:40.997218 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:40.997225 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:40.997477 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:40.998018 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:40.998023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:40.998026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:40.999083 (+ 1057us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:40.999609 (+ 526us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:41.000563 (+ 954us) server_negotiation.cc:299] Negotiation successful
0810 23:31:41.001006 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2089,"thread_start_us":1807,"threads_started":1}
I20260810 23:31:41.004431 9552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833a5c64818042678a463cba54817429" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40155 } }
I20260810 23:31:41.005417 9552 leader_election.cc:302] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 833a5c64818042678a463cba54817429; no voters:
I20260810 23:31:41.006300 9552 leader_election.cc:288] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:31:41.006577 9563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:41.007795 9563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [term 1 LEADER]: Becoming Leader. State: Replica: 833a5c64818042678a463cba54817429, State: Running, Role: LEADER
I20260810 23:31:41.008693 9563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833a5c64818042678a463cba54817429" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40155 } }
I20260810 23:31:41.009457 9552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:41.013828 9564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "833a5c64818042678a463cba54817429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833a5c64818042678a463cba54817429" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40155 } } }
I20260810 23:31:41.014329 9564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:41.014935 9565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 833a5c64818042678a463cba54817429. Latest consensus state: current_term: 1 leader_uuid: "833a5c64818042678a463cba54817429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833a5c64818042678a463cba54817429" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40155 } } }
I20260810 23:31:41.015326 9565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:41.017566 9576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:31:41.025875 9576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:31:41.041390 9576 catalog_manager.cc:1347] Generated new cluster ID: 9b6f25fcc72e4687b316fa5d0b4d14bd
I20260810 23:31:41.041564 9576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:31:41.085968 9576 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:31:41.087523 9576 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:31:41.097543 9576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 833a5c64818042678a463cba54817429: Generated new TSK 0
I20260810 23:31:41.098860 9576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:31:41.160591 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40155
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:41.514912 9595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:41.515177 9595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:41.515354 9595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:41.522629 9595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:41.522732 9595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:41.522941 9595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:31:41.533216 9595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40155
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:41.534518 9595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:41.535926 9595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:41.550482 9606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:41.550467 9605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:41.550526 9608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:41.551292 9595 server_base.cc:1034] running on GCE node
I20260810 23:31:41.551952 9595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:41.552932 9595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:41.554181 9595 hybrid_clock.cc:648] HybridClock initialized: now 1786404701554126 us; error 35 us; skew 500 ppm
I20260810 23:31:41.554598 9595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:41 dist-test-slave-04bb krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404701, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:41.558174 9595 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:31:41.559923 9595 webserver.cc:460] Webserver started at http://127.9.48.193:44899/ using document root <none> and password file <none>
I20260810 23:31:41.560734 9595 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:41.560876 9595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:41.561235 9595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:41.564302 9595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "a7a8469c24c3473c8afd659d0ad63a6b"
format_stamp: "Formatted at 2026-08-10 23:31:41 on dist-test-slave-04bb"
server_key: "070ead9e9335038892b3a4918f984110"
server_key_iv: "bc11ee350561178fd8521b3fd45ff7d1"
server_key_version: "encryptionkey@0"
I20260810 23:31:41.565199 9595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "a7a8469c24c3473c8afd659d0ad63a6b"
format_stamp: "Formatted at 2026-08-10 23:31:41 on dist-test-slave-04bb"
server_key: "070ead9e9335038892b3a4918f984110"
server_key_iv: "bc11ee350561178fd8521b3fd45ff7d1"
server_key_version: "encryptionkey@0"
I20260810 23:31:41.570554 9595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 23:31:41.574330 9619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:41.575742 9595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:31:41.575948 9595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "a7a8469c24c3473c8afd659d0ad63a6b"
format_stamp: "Formatted at 2026-08-10 23:31:41 on dist-test-slave-04bb"
server_key: "070ead9e9335038892b3a4918f984110"
server_key_iv: "bc11ee350561178fd8521b3fd45ff7d1"
server_key_version: "encryptionkey@0"
I20260810 23:31:41.576167 9595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:41.601977 9595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:41.606292 9595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:41.606597 9595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:41.607817 9595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:31:41.609440 9595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:31:41.609545 9595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:41.609773 9595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:31:41.609861 9595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:41.652413 9595 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:34853
I20260810 23:31:41.652515 9809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:34853 every 8 connection(s)
I20260810 23:31:41.653826 9595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:31:41.662508 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 9595
I20260810 23:31:41.662801 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:31:41.663699 9411 external_mini_cluster.cc:1428] Setting key 2d2487b4b91f29a2b8998ebba5b26b3a
Aug 10 23:31:41 dist-test-slave-04bb krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404701, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:41.678788 9816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:41.657452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:58903 (local address 127.9.48.254:40155)
0810 23:31:41.658029 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:41.658034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:41.659133 (+ 1099us) server_negotiation.cc:407] Connection header received
0810 23:31:41.661301 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:41.661306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:41.661376 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:41.661476 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:41.663819 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:41.664753 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:41.666581 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:41.666978 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:41.667131 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:41.671458 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:41.671467 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:41.671474 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:41.671516 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:41.675082 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:41.675866 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:41.675875 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:41.675882 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:41.676042 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:41.676549 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:41.676557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:41.676563 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:41.677015 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:41.677177 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:41.678010 (+ 833us) server_negotiation.cc:299] Negotiation successful
0810 23:31:41.678275 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":316,"threads_started":1}
I20260810 23:31:41.679641 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:41.658095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40155 (local address 127.9.48.193:58903)
0810 23:31:41.658942 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:41.658984 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:41.661001 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:41.661704 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:41.661713 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:41.662617 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:41.663588 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:41.663615 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:41.664911 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:41.664919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:41.666415 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:41.666433 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:41.667241 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:41.667251 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:41.667511 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:41.668320 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:41.668343 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:41.671212 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:41.675244 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:41.675251 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:41.675265 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:41.675732 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:41.676196 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:41.676200 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:41.676203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:41.676398 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:41.677161 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:41.677168 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:41.677510 (+ 342us) client_negotiation.cc:769] Sending connection context
0810 23:31:41.677863 (+ 353us) client_negotiation.cc:240] Negotiation successful
0810 23:31:41.678326 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":364,"threads_started":1}
I20260810 23:31:41.681663 9810 heartbeater.cc:347] Connected to a master server at 127.9.48.254:40155
I20260810 23:31:41.682267 9810 heartbeater.cc:464] Registering TS with master...
I20260810 23:31:41.683454 9810 heartbeater.cc:511] Master 127.9.48.254:40155 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
I20260810 23:31:41.687899 9493 ts_manager.cc:194] Registered new tserver with Master: a7a8469c24c3473c8afd659d0ad63a6b (127.9.48.193:34853)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
I20260810 23:31:41.690244 9493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:58903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:31:41.712733 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40155
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:42.058430 9824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:42.058768 9824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:42.058948 9824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:42.066365 9824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:42.066521 9824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:42.066716 9824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:31:42.076681 9824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40155
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:42.078064 9824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:42.079443 9824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:42.094218 9831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:42.094259 9832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:42.094305 9836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:42.098335 9824 server_base.cc:1034] running on GCE node
I20260810 23:31:42.099211 9824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:42.100567 9824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:42.101760 9824 hybrid_clock.cc:648] HybridClock initialized: now 1786404702101694 us; error 61 us; skew 500 ppm
I20260810 23:31:42.102155 9824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:42 dist-test-slave-04bb krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404702, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:42.105659 9824 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:31:42.107223 9824 webserver.cc:460] Webserver started at http://127.9.48.194:40981/ using document root <none> and password file <none>
I20260810 23:31:42.107977 9824 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:42.108129 9824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:42.108466 9824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:42.112663 9824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "419a84b63f754c43b78aa4a32fbc7d67"
format_stamp: "Formatted at 2026-08-10 23:31:42 on dist-test-slave-04bb"
server_key: "829855a9babac8408dfd308853cf9fb0"
server_key_iv: "93249e3a9674e4f2a9843d177f0359a9"
server_key_version: "encryptionkey@0"
I20260810 23:31:42.113497 9824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "419a84b63f754c43b78aa4a32fbc7d67"
format_stamp: "Formatted at 2026-08-10 23:31:42 on dist-test-slave-04bb"
server_key: "829855a9babac8408dfd308853cf9fb0"
server_key_iv: "93249e3a9674e4f2a9843d177f0359a9"
server_key_version: "encryptionkey@0"
I20260810 23:31:42.118932 9824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260810 23:31:42.122493 9851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:42.123885 9824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:31:42.124089 9824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "419a84b63f754c43b78aa4a32fbc7d67"
format_stamp: "Formatted at 2026-08-10 23:31:42 on dist-test-slave-04bb"
server_key: "829855a9babac8408dfd308853cf9fb0"
server_key_iv: "93249e3a9674e4f2a9843d177f0359a9"
server_key_version: "encryptionkey@0"
I20260810 23:31:42.124333 9824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:42.159152 9824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:42.169853 9824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:42.170173 9824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:42.171222 9824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:31:42.172713 9824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:31:42.172866 9824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:42.173020 9824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:31:42.173101 9824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:42.213985 9824 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:40311
I20260810 23:31:42.214085 10039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:40311 every 8 connection(s)
I20260810 23:31:42.215473 9824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:31:42.223064 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 9824
I20260810 23:31:42.223336 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:31:42.223799 9411 external_mini_cluster.cc:1428] Setting key a8b27f839090e26aa7d71aa279e5b59a
Aug 10 23:31:42 dist-test-slave-04bb krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404702, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:42.238643 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:42.218521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:54395 (local address 127.9.48.254:40155)
0810 23:31:42.218987 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:42.218992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:42.220163 (+ 1171us) server_negotiation.cc:407] Connection header received
0810 23:31:42.222371 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:42.222378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:42.222482 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:42.222621 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:42.224488 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:42.225376 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:42.227171 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:42.227709 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:42.227884 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:42.231879 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:42.231885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:42.231889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:42.231925 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:42.234911 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:42.235971 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:42.235975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:42.235977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:42.236116 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:42.236669 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:42.236673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:42.236675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:42.236990 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:42.237111 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:42.238110 (+ 999us) server_negotiation.cc:299] Negotiation successful
0810 23:31:42.238352 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":244,"threads_started":1}
I20260810 23:31:42.240294 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:42.219163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40155 (local address 127.9.48.194:54395)
0810 23:31:42.219979 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:42.220022 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:42.222051 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:42.222890 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:42.222900 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:42.223599 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:42.224291 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:42.224311 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:42.225546 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:42.225557 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:42.226965 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:42.226979 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:42.228004 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:42.228014 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:42.228335 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:42.229178 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:42.229209 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:42.231619 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:42.235075 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:42.235088 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:42.235109 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:42.235811 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:42.236241 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:42.236250 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:42.236257 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:42.236547 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:42.237134 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:42.237145 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:42.237631 (+ 486us) client_negotiation.cc:769] Sending connection context
0810 23:31:42.237974 (+ 343us) client_negotiation.cc:240] Negotiation successful
0810 23:31:42.238616 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":376,"threads_started":1}
I20260810 23:31:42.241910 10040 heartbeater.cc:347] Connected to a master server at 127.9.48.254:40155
I20260810 23:31:42.242241 10040 heartbeater.cc:464] Registering TS with master...
I20260810 23:31:42.243005 10040 heartbeater.cc:511] Master 127.9.48.254:40155 requested a full tablet report, sending...
I20260810 23:31:42.244493 9493 ts_manager.cc:194] Registered new tserver with Master: 419a84b63f754c43b78aa4a32fbc7d67 (127.9.48.194:40311)
I20260810 23:31:42.245358 9493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:54395
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:31:42.276495 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40155
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:42.625355 10052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:42.625677 10052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:42.625785 10052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:42.633231 10052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:42.633339 10052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:42.633458 10052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:31:42.643693 10052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40155
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:42.645093 10052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:42.646430 10052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:42.660845 10061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:42.660750 10062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:42.660818 10067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:42.661628 10052 server_base.cc:1034] running on GCE node
I20260810 23:31:42.662240 10052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:42.663133 10052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:42.664364 10052 hybrid_clock.cc:648] HybridClock initialized: now 1786404702664307 us; error 48 us; skew 500 ppm
I20260810 23:31:42.664768 10052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:42 dist-test-slave-04bb krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404702, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:42.668437 10052 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:31:42.670104 10052 webserver.cc:460] Webserver started at http://127.9.48.195:44099/ using document root <none> and password file <none>
I20260810 23:31:42.670826 10052 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:42.670982 10052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:42.671311 10052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:42.674415 10052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "9d6a0ceb09e64a3ca424fcd39c384646"
format_stamp: "Formatted at 2026-08-10 23:31:42 on dist-test-slave-04bb"
server_key: "a7a57ac0b9ae21acd1b4ebece41db0ce"
server_key_iv: "c34d2efe817ba7fdfcaa6196988556eb"
server_key_version: "encryptionkey@0"
I20260810 23:31:42.675205 10052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "9d6a0ceb09e64a3ca424fcd39c384646"
format_stamp: "Formatted at 2026-08-10 23:31:42 on dist-test-slave-04bb"
server_key: "a7a57ac0b9ae21acd1b4ebece41db0ce"
server_key_iv: "c34d2efe817ba7fdfcaa6196988556eb"
server_key_version: "encryptionkey@0"
I20260810 23:31:42.680748 10052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260810 23:31:42.684152 10079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:42.685611 10052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:31:42.685786 10052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "9d6a0ceb09e64a3ca424fcd39c384646"
format_stamp: "Formatted at 2026-08-10 23:31:42 on dist-test-slave-04bb"
server_key: "a7a57ac0b9ae21acd1b4ebece41db0ce"
server_key_iv: "c34d2efe817ba7fdfcaa6196988556eb"
server_key_version: "encryptionkey@0"
I20260810 23:31:42.685997 10052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:42.693761 9810 heartbeater.cc:503] Master 127.9.48.254:40155 was elected leader, sending a full tablet report...
I20260810 23:31:42.697917 10052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:42.701968 10052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:42.702262 10052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:42.703172 10052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:31:42.704602 10052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:31:42.704715 10052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:42.704860 10052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:31:42.704934 10052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:42.742844 10052 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:37561
I20260810 23:31:42.742946 10242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:37561 every 8 connection(s)
I20260810 23:31:42.744294 10052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:31:42.746749 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 10052
I20260810 23:31:42.746950 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:31:42.747571 9411 external_mini_cluster.cc:1428] Setting key 8d8f50ea93840b86fb9ec1c6ce379ae4
Aug 10 23:31:42 dist-test-slave-04bb krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404702, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:42.765776 10246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:42.747217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:36699 (local address 127.9.48.254:40155)
0810 23:31:42.747694 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:42.747700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:42.748924 (+ 1224us) server_negotiation.cc:407] Connection header received
0810 23:31:42.751345 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:42.751351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:42.751436 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:42.751554 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:42.753401 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:42.754191 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:42.755724 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:42.756116 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:42.756230 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:42.759618 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:42.759624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:42.759627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:42.759675 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:42.762451 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:42.763272 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:42.763277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:42.763279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:42.763443 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:42.763912 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:42.763916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:42.763918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:42.764201 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:42.764376 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:42.765146 (+ 770us) server_negotiation.cc:299] Negotiation successful
0810 23:31:42.765371 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":223,"threads_started":1}
I20260810 23:31:42.766706 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:42.747845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40155 (local address 127.9.48.195:36699)
0810 23:31:42.748705 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:42.748757 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:42.751054 (+ 2297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:42.751794 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:42.751801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:42.752667 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:42.753234 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:42.753252 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:42.754335 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:42.754339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:42.755582 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:42.755598 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:42.756330 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:42.756335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:42.756601 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:42.757240 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:42.757262 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:42.759427 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:42.762610 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:42.762618 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:42.762633 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:42.763121 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:42.763567 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:42.763575 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:42.763578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:42.763762 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:42.764338 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:42.764347 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:42.764752 (+ 405us) client_negotiation.cc:769] Sending connection context
0810 23:31:42.765018 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 23:31:42.765533 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":345,"threads_started":1}
I20260810 23:31:42.768122 10243 heartbeater.cc:347] Connected to a master server at 127.9.48.254:40155
I20260810 23:31:42.768429 10243 heartbeater.cc:464] Registering TS with master...
I20260810 23:31:42.769137 10243 heartbeater.cc:511] Master 127.9.48.254:40155 requested a full tablet report, sending...
I20260810 23:31:42.770479 9493 ts_manager.cc:194] Registered new tserver with Master: 9d6a0ceb09e64a3ca424fcd39c384646 (127.9.48.195:37561)
I20260810 23:31:42.771256 9493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:36699
I20260810 23:31:42.777047 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:31:42 dist-test-slave-04bb krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404702, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:31:42 dist-test-slave-04bb krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404702, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:31:42.830060 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:42.813876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40812 (local address 127.9.48.193:34853)
0810 23:31:42.814554 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:42.814567 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:42.814601 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:31:42.814686 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:42.814693 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:42.814923 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:42.815101 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:42.816650 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:42.817778 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:42.819256 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:42.819829 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:42.819980 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:42.824212 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:42.824224 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:42.824240 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:42.824282 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:42.827148 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:42.827897 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:42.827904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:42.827906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:42.828035 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:42.828426 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:42.828432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:42.828440 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:42.828927 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:42.829185 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:42.829437 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 23:31:42.829706 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":323,"threads_started":1}
W20260810 23:31:42.831483 9661 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40812
Aug 10 23:31:42 dist-test-slave-04bb krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:31:42 dist-test-slave-04bb krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:31:42.882192 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:42.867443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40822 (local address 127.9.48.193:34853)
0810 23:31:42.867691 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:42.867699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:42.868002 (+ 303us) server_negotiation.cc:407] Connection header received
0810 23:31:42.868232 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:42.868238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:42.868345 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:42.868457 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:42.869875 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:42.870659 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:42.871931 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:42.872351 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:42.872490 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:42.876734 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:42.876741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:42.876745 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:42.876782 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:42.879438 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:42.880278 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:42.880283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:42.880286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:42.880423 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:42.880872 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:42.880878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:42.880882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:42.881242 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:42.881438 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:42.881782 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 23:31:42.881957 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:31:42.884394 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 9595
I20260810 23:31:42.897866 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:42.885754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.9.48.193:34853)
0810 23:31:42.885922 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:42.885926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:42.886134 (+ 208us) server_negotiation.cc:407] Connection header received
0810 23:31:42.886236 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:42.886239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:42.886296 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:42.886385 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:42.887626 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:42.888395 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:42.889888 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:42.890599 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:42.890752 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:42.893078 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:42.893083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:42.893086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:42.893117 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:42.895744 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:42.896293 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:42.896297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:42.896299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:42.896430 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:42.896807 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:42.896811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:42.896815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:42.897053 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:42.897245 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:42.897445 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 23:31:42.897593 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 23:31:42.898710 9804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:42.988709 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 9595
I20260810 23:31:43.015076 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 9824
Aug 10 23:31:43 dist-test-slave-04bb krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:31:43.032022 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:43.016056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.9.48.194:40311)
0810 23:31:43.016699 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:43.016706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:43.016734 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:31:43.016819 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:43.016831 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:43.017079 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:43.017287 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:43.018610 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:43.019614 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:43.021028 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:43.021578 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:43.021731 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:43.025855 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:43.025867 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:43.025881 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:43.025921 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:43.029074 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:43.029787 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:43.029794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:43.029797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:43.029922 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:43.030333 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:43.030339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:43.030344 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:43.030690 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:43.031018 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:43.031353 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 23:31:43.031628 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":291,"threads_started":1}
I20260810 23:31:43.033528 10035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:43.120749 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 9824
I20260810 23:31:43.146672 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 10052
Aug 10 23:31:43 dist-test-slave-04bb krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:31:43.163533 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:43.147708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37730 (local address 127.9.48.195:37561)
0810 23:31:43.148440 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:43.148447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:43.148480 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:31:43.148622 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:43.148629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:43.148866 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:43.149047 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:43.150415 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:43.151490 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:43.152971 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:43.153551 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:43.153718 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:43.157990 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:43.158001 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:43.158016 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:43.158050 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:43.160944 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:43.161646 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:43.161653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:43.161656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:43.161795 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:43.162126 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:43.162129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:43.162132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:43.162486 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:43.162734 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:43.163004 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 23:31:43.163242 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":386,"threads_started":1}
I20260810 23:31:43.165221 10237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:43.251037 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 10052
I20260810 23:31:43.278223 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 9440
I20260810 23:31:43.279663 9543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:43.384476 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 9440
2026-08-10T23:31:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4774 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:31:43 dist-test-slave-04bb krb5kdc[10282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:31:43 dist-test-slave-04bb krb5kdc[10282](info): set up 2 sockets
Aug 10 23:31:43 dist-test-slave-04bb krb5kdc[10282](info): commencing operation
krb5kdc: starting...
W20260810 23:31:44.691344 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.256s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:31:44 dist-test-slave-04bb krb5kdc[10282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:31:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:31:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:31:44.896940 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:35559
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:38855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:35559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:45.247211 10302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:45.247504 10302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:45.247611 10302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:45.254805 10302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:45.254917 10302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:45.254969 10302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:45.255025 10302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:45.255067 10302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:45.264698 10302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:38855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:35559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:35559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:45.265930 10302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:45.267275 10302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:45.280058 10308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:45.280121 10313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:45.280092 10310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:45.281126 10302 server_base.cc:1034] running on GCE node
I20260810 23:31:45.281929 10302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:45.283365 10302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:45.284634 10302 hybrid_clock.cc:648] HybridClock initialized: now 1786404705284584 us; error 73 us; skew 500 ppm
I20260810 23:31:45.284991 10302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:45 dist-test-slave-04bb krb5kdc[10282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404705, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:45.288192 10302 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:31:45.290007 10302 webserver.cc:460] Webserver started at http://127.9.48.254:44853/ using document root <none> and password file <none>
I20260810 23:31:45.290959 10302 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:45.291105 10302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:45.291457 10302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:45.294536 10302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "8a3c86ed33614cc6a706592915c407b1"
format_stamp: "Formatted at 2026-08-10 23:31:45 on dist-test-slave-04bb"
server_key: "663bd69f928fee0e6b7870890b378b38"
server_key_iv: "74dd2e7016d3f87ccf8b89188a2705c2"
server_key_version: "encryptionkey@0"
I20260810 23:31:45.295315 10302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "8a3c86ed33614cc6a706592915c407b1"
format_stamp: "Formatted at 2026-08-10 23:31:45 on dist-test-slave-04bb"
server_key: "663bd69f928fee0e6b7870890b378b38"
server_key_iv: "74dd2e7016d3f87ccf8b89188a2705c2"
server_key_version: "encryptionkey@0"
I20260810 23:31:45.300607 10302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:31:45.304054 10324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:45.305377 10302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:31:45.305560 10302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "8a3c86ed33614cc6a706592915c407b1"
format_stamp: "Formatted at 2026-08-10 23:31:45 on dist-test-slave-04bb"
server_key: "663bd69f928fee0e6b7870890b378b38"
server_key_iv: "74dd2e7016d3f87ccf8b89188a2705c2"
server_key_version: "encryptionkey@0"
I20260810 23:31:45.305754 10302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:45.339763 10302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:45.344079 10302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:45.344400 10302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:45.363751 10302 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:35559
I20260810 23:31:45.363874 10400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:35559 every 8 connection(s)
I20260810 23:31:45.365267 10302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:31:45.368788 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 10302
I20260810 23:31:45.369033 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:31:45.369346 10401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:45.369776 9411 external_mini_cluster.cc:1428] Setting key 4c11fcb5b8a5c42441525aa3211da112
I20260810 23:31:45.384197 10401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1: Bootstrap starting.
Aug 10 23:31:45 dist-test-slave-04bb krb5kdc[10282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:45.387710 10401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:45.388931 10401 log.cc:824] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:45.391724 10401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1: No bootstrap required, opened a new log
I20260810 23:31:45.393355 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:45.371770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.9.48.254:35559)
0810 23:31:45.372655 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:45.372666 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:45.372710 (+ 44us) server_negotiation.cc:407] Connection header received
0810 23:31:45.374420 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:45.374445 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:45.374846 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:45.375173 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:45.377110 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:45.378405 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:45.380178 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:45.380969 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:45.381108 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:45.385942 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:45.385950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:45.385964 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:45.386003 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:45.389265 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:45.389988 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:45.389994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:45.390000 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:45.390163 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:45.390561 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:45.390566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:45.390569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:45.391074 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:45.391405 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:45.391911 (+ 506us) server_negotiation.cc:299] Negotiation successful
0810 23:31:45.392324 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":394,"threads_started":1}
I20260810 23:31:45.400332 10401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3c86ed33614cc6a706592915c407b1" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 35559 } }
I20260810 23:31:45.400889 10401 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:45.401012 10401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3c86ed33614cc6a706592915c407b1, State: Initialized, Role: FOLLOWER
I20260810 23:31:45.401512 10401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3c86ed33614cc6a706592915c407b1" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 35559 } }
I20260810 23:31:45.401734 10401 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:31:45.401831 10401 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:31:45.401947 10401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:45.404443 10401 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3c86ed33614cc6a706592915c407b1" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 35559 } }
I20260810 23:31:45.404953 10401 leader_election.cc:302] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a3c86ed33614cc6a706592915c407b1; no voters:
I20260810 23:31:45.405470 10401 leader_election.cc:288] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:31:45.405620 10407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:45.406308 10407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [term 1 LEADER]: Becoming Leader. State: Replica: 8a3c86ed33614cc6a706592915c407b1, State: Running, Role: LEADER
I20260810 23:31:45.406868 10407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3c86ed33614cc6a706592915c407b1" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 35559 } }
I20260810 23:31:45.407529 10401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:45.410454 10409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a3c86ed33614cc6a706592915c407b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3c86ed33614cc6a706592915c407b1" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 35559 } } }
I20260810 23:31:45.410835 10409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:45.413247 10411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3c86ed33614cc6a706592915c407b1. Latest consensus state: current_term: 1 leader_uuid: "8a3c86ed33614cc6a706592915c407b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3c86ed33614cc6a706592915c407b1" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 35559 } } }
I20260810 23:31:45.413942 10411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:45.414134 10420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:31:45.418876 10420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:31:45.429636 10420 catalog_manager.cc:1347] Generated new cluster ID: dbfe3988721540ff90b5f7a986edbb49
I20260810 23:31:45.429929 10420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:31:45.441128 10420 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:31:45.442365 10420 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:31:45.451602 10420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a3c86ed33614cc6a706592915c407b1: Generated new TSK 0
I20260810 23:31:45.452450 10420 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:31:45.502518 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:35559
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:38855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:45.843024 10439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:45.843297 10439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:45.843400 10439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:45.850647 10439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:45.850780 10439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:45.850900 10439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:31:45.860641 10439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:38855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:35559
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:45.861898 10439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:45.863232 10439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:45.877636 10453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:45.877561 10454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:45.877607 10457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:45.878793 10439 server_base.cc:1034] running on GCE node
I20260810 23:31:45.879400 10439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:45.880362 10439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:45.881572 10439 hybrid_clock.cc:648] HybridClock initialized: now 1786404705881519 us; error 50 us; skew 500 ppm
I20260810 23:31:45.881938 10439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:45 dist-test-slave-04bb krb5kdc[10282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404705, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:45.885289 10439 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:31:45.886829 10439 webserver.cc:460] Webserver started at http://127.9.48.193:36573/ using document root <none> and password file <none>
I20260810 23:31:45.887569 10439 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:45.887773 10439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:45.888147 10439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:45.891336 10439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "46385dda669f41eca466dc6d65488630"
format_stamp: "Formatted at 2026-08-10 23:31:45 on dist-test-slave-04bb"
server_key: "90c4d06c48785c19d18264318541578d"
server_key_iv: "ab07aa562ba373e463607a950e4eeb0d"
server_key_version: "encryptionkey@0"
I20260810 23:31:45.892170 10439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "46385dda669f41eca466dc6d65488630"
format_stamp: "Formatted at 2026-08-10 23:31:45 on dist-test-slave-04bb"
server_key: "90c4d06c48785c19d18264318541578d"
server_key_iv: "ab07aa562ba373e463607a950e4eeb0d"
server_key_version: "encryptionkey@0"
I20260810 23:31:45.897934 10439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:31:45.901552 10469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:45.902949 10439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:31:45.903082 10439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "46385dda669f41eca466dc6d65488630"
format_stamp: "Formatted at 2026-08-10 23:31:45 on dist-test-slave-04bb"
server_key: "90c4d06c48785c19d18264318541578d"
server_key_iv: "ab07aa562ba373e463607a950e4eeb0d"
server_key_version: "encryptionkey@0"
I20260810 23:31:45.903256 10439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:45.939425 10439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:45.943815 10439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:45.944058 10439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:45.945073 10439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:31:45.946434 10439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:31:45.946516 10439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:45.946609 10439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:31:45.946655 10439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:45.985375 10439 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:37985
I20260810 23:31:45.985646 10642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:37985 every 8 connection(s)
I20260810 23:31:45.986764 10439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:31:45.993114 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 10439
I20260810 23:31:45.993345 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:31:45.994130 9411 external_mini_cluster.cc:1428] Setting key baeefa4662527633fba84e1baf6b7da7
Aug 10 23:31:45 dist-test-slave-04bb krb5kdc[10282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404705, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:46.012811 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:45.989894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:46063 (local address 127.9.48.254:35559)
0810 23:31:45.990417 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:45.990422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:45.991642 (+ 1220us) server_negotiation.cc:407] Connection header received
0810 23:31:45.994024 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:45.994031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:45.994162 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:45.994304 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:45.996475 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:45.997655 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:45.999442 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:46.000165 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:46.000325 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:46.004582 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:46.004593 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:46.004602 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:46.004650 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:46.008997 (+ 4347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:46.009837 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:46.009846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:46.009853 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:46.010019 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:46.010536 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:46.010545 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:46.010553 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:46.010997 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:46.011182 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:46.011896 (+ 714us) server_negotiation.cc:299] Negotiation successful
0810 23:31:46.012221 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":298,"threads_started":1}
I20260810 23:31:46.013449 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:45.990570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:35559 (local address 127.9.48.193:46063)
0810 23:31:45.991435 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:45.991485 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:45.993701 (+ 2216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:45.994573 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:45.994582 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:45.995350 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:45.996249 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:45.996282 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:45.997859 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:45.997863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:45.999273 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:45.999286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:46.000424 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:46.000429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:46.000731 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:46.001499 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:46.001530 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:46.004309 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:46.009136 (+ 4827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:46.009143 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:46.009155 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:46.009714 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:46.010136 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:46.010140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:46.010142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:46.010407 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:46.011135 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:46.011141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:46.011467 (+ 326us) client_negotiation.cc:769] Sending connection context
0810 23:31:46.011713 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 23:31:46.012117 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":357,"threads_started":1}
I20260810 23:31:46.015146 10643 heartbeater.cc:347] Connected to a master server at 127.9.48.254:35559
I20260810 23:31:46.015532 10643 heartbeater.cc:464] Registering TS with master...
I20260810 23:31:46.016602 10643 heartbeater.cc:511] Master 127.9.48.254:35559 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
I20260810 23:31:46.019158 10352 ts_manager.cc:194] Registered new tserver with Master: 46385dda669f41eca466dc6d65488630 (127.9.48.193:37985)
I20260810 23:31:46.020944 10352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:46063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:31:46.047448 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:35559
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:38855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:46.414870 10655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:46.415185 10655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:46.415364 10655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:46.422612 10655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:46.422767 10655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:46.422950 10655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:31:46.432770 10655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:38855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:35559
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:46.434053 10655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:46.435453 10655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:46.448796 10667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:46.448866 10664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:46.448889 10663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:46.450596 10655 server_base.cc:1034] running on GCE node
I20260810 23:31:46.451376 10655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:46.452374 10655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:46.453600 10655 hybrid_clock.cc:648] HybridClock initialized: now 1786404706453551 us; error 73 us; skew 500 ppm
I20260810 23:31:46.453945 10655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:46 dist-test-slave-04bb krb5kdc[10282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404706, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:46.457284 10655 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:31:46.458766 10655 webserver.cc:460] Webserver started at http://127.9.48.194:42039/ using document root <none> and password file <none>
I20260810 23:31:46.459450 10655 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:46.459599 10655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:46.459924 10655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:46.462874 10655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "fa05a6aff81945758e6d20e64fa873a1"
format_stamp: "Formatted at 2026-08-10 23:31:46 on dist-test-slave-04bb"
server_key: "6ab9a80b75f2f47c3bf70fc5363267f4"
server_key_iv: "c5a41a1899236fd0919f24de1a6672b8"
server_key_version: "encryptionkey@0"
I20260810 23:31:46.463661 10655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "fa05a6aff81945758e6d20e64fa873a1"
format_stamp: "Formatted at 2026-08-10 23:31:46 on dist-test-slave-04bb"
server_key: "6ab9a80b75f2f47c3bf70fc5363267f4"
server_key_iv: "c5a41a1899236fd0919f24de1a6672b8"
server_key_version: "encryptionkey@0"
I20260810 23:31:46.468816 10655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 23:31:46.472265 10683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:46.473608 10655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:31:46.473780 10655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "fa05a6aff81945758e6d20e64fa873a1"
format_stamp: "Formatted at 2026-08-10 23:31:46 on dist-test-slave-04bb"
server_key: "6ab9a80b75f2f47c3bf70fc5363267f4"
server_key_iv: "c5a41a1899236fd0919f24de1a6672b8"
server_key_version: "encryptionkey@0"
I20260810 23:31:46.473984 10655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:46.498157 10655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:46.502264 10655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:46.502573 10655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:46.503588 10655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:31:46.504980 10655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:31:46.505060 10655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:46.505235 10655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:31:46.505290 10655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:46.540982 10655 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:38743
I20260810 23:31:46.541074 10864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:38743 every 8 connection(s)
I20260810 23:31:46.542330 10655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:31:46.546468 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 10655
I20260810 23:31:46.546686 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:31:46.547264 9411 external_mini_cluster.cc:1428] Setting key 409382215fd8de5611dd25ef1c184dde
Aug 10 23:31:46 dist-test-slave-04bb krb5kdc[10282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404706, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:46.564496 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:46.545092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:40279 (local address 127.9.48.254:35559)
0810 23:31:46.545547 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:46.545553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:46.546779 (+ 1226us) server_negotiation.cc:407] Connection header received
0810 23:31:46.549359 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:46.549365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:46.549451 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:46.549539 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:46.551074 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:46.552274 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:46.553936 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:46.554669 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:46.554830 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:46.558542 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:46.558547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:46.558550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:46.558583 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:46.561358 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:46.562101 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:46.562106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:46.562108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:46.562236 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:46.562685 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:46.562690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:46.562692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:46.562948 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:46.563112 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:46.563878 (+ 766us) server_negotiation.cc:299] Negotiation successful
0810 23:31:46.564106 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":224,"threads_started":1}
I20260810 23:31:46.565578 10873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:46.545711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:35559 (local address 127.9.48.194:40279)
0810 23:31:46.546581 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:46.546646 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:46.549033 (+ 2387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:46.549801 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:46.549816 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:46.550324 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:46.550891 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:46.550909 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:46.552427 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:46.552430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:46.553766 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:46.553778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:46.554897 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:46.554902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:46.555089 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:46.555779 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:46.555802 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:46.558311 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:46.561505 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:46.561514 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:46.561529 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:46.561961 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:46.562351 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:46.562356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:46.562360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:46.562554 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:46.563089 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:46.563097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:46.563492 (+ 395us) client_negotiation.cc:769] Sending connection context
0810 23:31:46.563766 (+ 274us) client_negotiation.cc:240] Negotiation successful
0810 23:31:46.564314 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":333,"threads_started":1}
I20260810 23:31:46.567173 10865 heartbeater.cc:347] Connected to a master server at 127.9.48.254:35559
I20260810 23:31:46.567536 10865 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
I20260810 23:31:46.568401 10865 heartbeater.cc:511] Master 127.9.48.254:35559 requested a full tablet report, sending...
I20260810 23:31:46.569967 10352 ts_manager.cc:194] Registered new tserver with Master: fa05a6aff81945758e6d20e64fa873a1 (127.9.48.194:38743)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
I20260810 23:31:46.571136 10352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:40279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:31:46.596354 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:35559
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:38855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:46.941728 10876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:46.942047 10876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:46.942232 10876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:46.949397 10876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:46.949553 10876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:46.949734 10876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:31:46.959491 10876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:38855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:35559
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:46.960911 10876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:46.962306 10876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:46.976230 10882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:46.976289 10883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:46.976330 10886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:46.977383 10876 server_base.cc:1034] running on GCE node
I20260810 23:31:46.978027 10876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:46.979293 10876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:46.980504 10876 hybrid_clock.cc:648] HybridClock initialized: now 1786404706980446 us; error 47 us; skew 500 ppm
I20260810 23:31:46.980886 10876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:46 dist-test-slave-04bb krb5kdc[10282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404706, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:46.984272 10876 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:31:46.986070 10876 webserver.cc:460] Webserver started at http://127.9.48.195:38199/ using document root <none> and password file <none>
I20260810 23:31:46.986773 10876 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:46.986922 10876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:46.987259 10876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:46.990290 10876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "7424e30f6b194f8991181ae6d42ce731"
format_stamp: "Formatted at 2026-08-10 23:31:46 on dist-test-slave-04bb"
server_key: "b94d071464d8df87e16b2f51356dc2d8"
server_key_iv: "e16aea0bfdecc7f91289869297038b6f"
server_key_version: "encryptionkey@0"
I20260810 23:31:46.991112 10876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "7424e30f6b194f8991181ae6d42ce731"
format_stamp: "Formatted at 2026-08-10 23:31:46 on dist-test-slave-04bb"
server_key: "b94d071464d8df87e16b2f51356dc2d8"
server_key_iv: "e16aea0bfdecc7f91289869297038b6f"
server_key_version: "encryptionkey@0"
I20260810 23:31:46.996613 10876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:31:47.000099 10895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:47.001437 10876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:31:47.001616 10876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "7424e30f6b194f8991181ae6d42ce731"
format_stamp: "Formatted at 2026-08-10 23:31:46 on dist-test-slave-04bb"
server_key: "b94d071464d8df87e16b2f51356dc2d8"
server_key_iv: "e16aea0bfdecc7f91289869297038b6f"
server_key_version: "encryptionkey@0"
I20260810 23:31:47.001811 10876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:47.021311 10876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:47.024003 10643 heartbeater.cc:503] Master 127.9.48.254:35559 was elected leader, sending a full tablet report...
I20260810 23:31:47.026813 10876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:47.027148 10876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:47.028267 10876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:31:47.029937 10876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:31:47.030057 10876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:47.030259 10876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:31:47.030350 10876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:47.070499 10876 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:33543
I20260810 23:31:47.070596 11074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:33543 every 8 connection(s)
I20260810 23:31:47.071913 10876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:31:47.077836 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 10876
I20260810 23:31:47.078101 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:31:47.078794 9411 external_mini_cluster.cc:1428] Setting key 93672d3e4ef2f5adcb41057b1f47e8f2
Aug 10 23:31:47 dist-test-slave-04bb krb5kdc[10282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404706, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:47.094719 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.074942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:40767 (local address 127.9.48.254:35559)
0810 23:31:47.075445 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:47.075452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:47.076472 (+ 1020us) server_negotiation.cc:407] Connection header received
0810 23:31:47.078625 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:47.078631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:47.078742 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:47.078873 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:47.080745 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.081890 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.083774 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.084150 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.084246 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.087910 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:47.087914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:47.087918 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:47.087949 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:47.091115 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.091933 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.091939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.091942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.092076 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.092582 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.092587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.092591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.092880 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:47.093060 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:47.094091 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0810 23:31:47.094279 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":236,"threads_started":1}
I20260810 23:31:47.095741 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.075430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:35559 (local address 127.9.48.195:40767)
0810 23:31:47.076211 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:47.076259 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:47.078287 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:47.079139 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:47.079149 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:47.079742 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:47.080514 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:47.080586 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.082080 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.082086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:47.083616 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:47.083629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.084381 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.084387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:47.084665 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.085412 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:47.085440 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:47.087672 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:47.091257 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:47.091266 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:47.091281 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:47.091805 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:47.092190 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:47.092194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:47.092198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:47.092442 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:47.093040 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:47.093071 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:47.093608 (+ 537us) client_negotiation.cc:769] Sending connection context
0810 23:31:47.093968 (+ 360us) client_negotiation.cc:240] Negotiation successful
0810 23:31:47.094512 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":314,"threads_started":1}
I20260810 23:31:47.097270 11075 heartbeater.cc:347] Connected to a master server at 127.9.48.254:35559
I20260810 23:31:47.097605 11075 heartbeater.cc:464] Registering TS with master...
I20260810 23:31:47.098415 11075 heartbeater.cc:511] Master 127.9.48.254:35559 requested a full tablet report, sending...
I20260810 23:31:47.099900 10352 ts_manager.cc:194] Registered new tserver with Master: 7424e30f6b194f8991181ae6d42ce731 (127.9.48.195:33543)
I20260810 23:31:47.100876 10352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:40767
I20260810 23:31:47.107400 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:31:47.129194 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.115663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.9.48.254:35559)
0810 23:31:47.115845 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:47.115852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:47.116386 (+ 534us) server_negotiation.cc:407] Connection header received
0810 23:31:47.116644 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:47.116651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:47.116721 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:47.116831 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:47.118477 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.119199 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.120471 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.120949 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.121086 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.123897 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:47.123907 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:47.123910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:47.123941 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:47.126496 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.127136 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.127145 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.127150 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.127290 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.127673 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.127679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.127683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.128016 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:47.128283 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:47.128654 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 23:31:47.128895 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:31:47.137539 10352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:31:47.145850 10352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:31:47.174918 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.161849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.9.48.194:38743)
0810 23:31:47.164145 (+ 2296us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:47.164160 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:47.164200 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:31:47.164336 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:47.164348 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:47.164742 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:47.166778 (+ 2036us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:47.167087 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.168902 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.171926 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.173101 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.173259 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.173485 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:47.174318 (+ 833us) server_negotiation.cc:299] Negotiation successful
0810 23:31:47.174527 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2060,"thread_start_us":1892,"threads_started":1}
I20260810 23:31:47.175563 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.161374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:38743 (local address 127.0.0.1:43734)
0810 23:31:47.163239 (+ 1865us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:47.163262 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:47.163390 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:47.165936 (+ 2546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:47.165947 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:47.165997 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:47.166521 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:47.166539 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.169079 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.169084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:47.171759 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:47.171772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.173674 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.173682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:47.173897 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.173951 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 23:31:47.174091 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 23:31:47.174296 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1746,"mutex_wait_us":73,"thread_start_us":3788,"threads_started":1}
I20260810 23:31:47.175900 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.161204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:33543 (local address 127.0.0.1:42682)
0810 23:31:47.162136 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:47.162189 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:47.162426 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:47.168046 (+ 5620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:47.168051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:47.168079 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:47.168619 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:47.168635 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.170648 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.170661 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:47.172946 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:47.172960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.174861 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.174873 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:47.175050 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.175098 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 23:31:47.175210 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 23:31:47.175314 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":4,"thread_start_us":532,"threads_started":1}
I20260810 23:31:47.176117 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.161264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.9.48.195:33543)
0810 23:31:47.167431 (+ 6167us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:47.167440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:47.167476 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:31:47.167577 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:47.167585 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:47.167814 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:47.168029 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:47.168861 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.170482 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.173118 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.174566 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.174821 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.175083 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:47.175562 (+ 479us) server_negotiation.cc:299] Negotiation successful
0810 23:31:47.175762 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5979,"thread_start_us":5764,"threads_started":1}
I20260810 23:31:47.177253 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.163178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.9.48.193:37985)
0810 23:31:47.167013 (+ 3835us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:47.167027 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:47.167062 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:31:47.167162 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:47.167168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:47.167401 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:47.167600 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:47.168335 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.170084 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.173082 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.174715 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.174880 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.175127 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:47.176692 (+ 1565us) server_negotiation.cc:299] Negotiation successful
0810 23:31:47.176860 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3615,"thread_start_us":3423,"threads_started":1}
I20260810 23:31:47.177300 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.162657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:37985 (local address 127.0.0.1:33100)
0810 23:31:47.165513 (+ 2856us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:47.165538 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:47.165694 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:47.167593 (+ 1899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:47.167597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:47.167624 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:47.168088 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:47.168100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.170256 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.170262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:47.172891 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:47.172907 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.175239 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.175246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:47.176441 (+ 1195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.176468 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:31:47.176623 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 23:31:47.176819 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2684,"mutex_wait_us":80,"thread_start_us":728,"threads_started":1}
I20260810 23:31:47.180861 10969 tablet_service.cc:1467] Processing CreateTablet for tablet b53d146a30d640e9a56269d54df8c8f7 (DEFAULT_TABLE table=test-table [id=f9fa3a3891414aeb878e03c6f24d8d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:47.181253 10540 tablet_service.cc:1467] Processing CreateTablet for tablet b53d146a30d640e9a56269d54df8c8f7 (DEFAULT_TABLE table=test-table [id=f9fa3a3891414aeb878e03c6f24d8d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:47.181757 10762 tablet_service.cc:1467] Processing CreateTablet for tablet b53d146a30d640e9a56269d54df8c8f7 (DEFAULT_TABLE table=test-table [id=f9fa3a3891414aeb878e03c6f24d8d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:47.182467 10969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b53d146a30d640e9a56269d54df8c8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:47.182766 10540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b53d146a30d640e9a56269d54df8c8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:47.183082 10762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b53d146a30d640e9a56269d54df8c8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:47.199021 11102 tablet_bootstrap.cc:492] T b53d146a30d640e9a56269d54df8c8f7 P 46385dda669f41eca466dc6d65488630: Bootstrap starting.
I20260810 23:31:47.200712 11103 tablet_bootstrap.cc:492] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1: Bootstrap starting.
I20260810 23:31:47.205348 11101 tablet_bootstrap.cc:492] T b53d146a30d640e9a56269d54df8c8f7 P 7424e30f6b194f8991181ae6d42ce731: Bootstrap starting.
I20260810 23:31:47.206949 11103 tablet_bootstrap.cc:654] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:47.207123 11102 tablet_bootstrap.cc:654] T b53d146a30d640e9a56269d54df8c8f7 P 46385dda669f41eca466dc6d65488630: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:47.209475 11103 log.cc:824] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:47.209637 11102 log.cc:824] T b53d146a30d640e9a56269d54df8c8f7 P 46385dda669f41eca466dc6d65488630: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:47.211205 11101 tablet_bootstrap.cc:654] T b53d146a30d640e9a56269d54df8c8f7 P 7424e30f6b194f8991181ae6d42ce731: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:47.213805 11101 log.cc:824] T b53d146a30d640e9a56269d54df8c8f7 P 7424e30f6b194f8991181ae6d42ce731: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:47.215523 11103 tablet_bootstrap.cc:492] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1: No bootstrap required, opened a new log
I20260810 23:31:47.215723 11102 tablet_bootstrap.cc:492] T b53d146a30d640e9a56269d54df8c8f7 P 46385dda669f41eca466dc6d65488630: No bootstrap required, opened a new log
I20260810 23:31:47.216097 11103 ts_tablet_manager.cc:1397] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20260810 23:31:47.216138 11102 ts_tablet_manager.cc:1397] T b53d146a30d640e9a56269d54df8c8f7 P 46385dda669f41eca466dc6d65488630: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.013s
I20260810 23:31:47.219442 11101 tablet_bootstrap.cc:492] T b53d146a30d640e9a56269d54df8c8f7 P 7424e30f6b194f8991181ae6d42ce731: No bootstrap required, opened a new log
I20260810 23:31:47.220000 11101 ts_tablet_manager.cc:1397] T b53d146a30d640e9a56269d54df8c8f7 P 7424e30f6b194f8991181ae6d42ce731: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20260810 23:31:47.228693 11102 raft_consensus.cc:348] T b53d146a30d640e9a56269d54df8c8f7 P 46385dda669f41eca466dc6d65488630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7424e30f6b194f8991181ae6d42ce731" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33543 } } peers { permanent_uuid: "46385dda669f41eca466dc6d65488630" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37985 } } peers { permanent_uuid: "fa05a6aff81945758e6d20e64fa873a1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38743 } }
I20260810 23:31:47.229298 11102 raft_consensus.cc:374] T b53d146a30d640e9a56269d54df8c8f7 P 46385dda669f41eca466dc6d65488630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:47.229401 11102 raft_consensus.cc:729] T b53d146a30d640e9a56269d54df8c8f7 P 46385dda669f41eca466dc6d65488630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46385dda669f41eca466dc6d65488630, State: Initialized, Role: FOLLOWER
I20260810 23:31:47.230142 11102 consensus_queue.cc:260] T b53d146a30d640e9a56269d54df8c8f7 P 46385dda669f41eca466dc6d65488630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7424e30f6b194f8991181ae6d42ce731" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33543 } } peers { permanent_uuid: "46385dda669f41eca466dc6d65488630" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37985 } } peers { permanent_uuid: "fa05a6aff81945758e6d20e64fa873a1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38743 } }
I20260810 23:31:47.234567 11103 raft_consensus.cc:348] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7424e30f6b194f8991181ae6d42ce731" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33543 } } peers { permanent_uuid: "46385dda669f41eca466dc6d65488630" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37985 } } peers { permanent_uuid: "fa05a6aff81945758e6d20e64fa873a1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38743 } }
I20260810 23:31:47.235126 11103 raft_consensus.cc:374] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:47.235240 11103 raft_consensus.cc:729] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa05a6aff81945758e6d20e64fa873a1, State: Initialized, Role: FOLLOWER
I20260810 23:31:47.235312 11102 ts_tablet_manager.cc:1428] T b53d146a30d640e9a56269d54df8c8f7 P 46385dda669f41eca466dc6d65488630: Time spent starting tablet: real 0.019s user 0.008s sys 0.012s
I20260810 23:31:47.236153 11103 consensus_queue.cc:260] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7424e30f6b194f8991181ae6d42ce731" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33543 } } peers { permanent_uuid: "46385dda669f41eca466dc6d65488630" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37985 } } peers { permanent_uuid: "fa05a6aff81945758e6d20e64fa873a1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38743 } }
I20260810 23:31:47.238144 10865 heartbeater.cc:503] Master 127.9.48.254:35559 was elected leader, sending a full tablet report...
I20260810 23:31:47.238380 11103 ts_tablet_manager.cc:1428] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1: Time spent starting tablet: real 0.022s user 0.012s sys 0.007s
I20260810 23:31:47.238233 11101 raft_consensus.cc:348] T b53d146a30d640e9a56269d54df8c8f7 P 7424e30f6b194f8991181ae6d42ce731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7424e30f6b194f8991181ae6d42ce731" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33543 } } peers { permanent_uuid: "46385dda669f41eca466dc6d65488630" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37985 } } peers { permanent_uuid: "fa05a6aff81945758e6d20e64fa873a1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38743 } }
I20260810 23:31:47.238739 11101 raft_consensus.cc:374] T b53d146a30d640e9a56269d54df8c8f7 P 7424e30f6b194f8991181ae6d42ce731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:47.238855 11101 raft_consensus.cc:729] T b53d146a30d640e9a56269d54df8c8f7 P 7424e30f6b194f8991181ae6d42ce731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7424e30f6b194f8991181ae6d42ce731, State: Initialized, Role: FOLLOWER
I20260810 23:31:47.239609 11101 consensus_queue.cc:260] T b53d146a30d640e9a56269d54df8c8f7 P 7424e30f6b194f8991181ae6d42ce731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7424e30f6b194f8991181ae6d42ce731" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33543 } } peers { permanent_uuid: "46385dda669f41eca466dc6d65488630" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37985 } } peers { permanent_uuid: "fa05a6aff81945758e6d20e64fa873a1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38743 } }
W20260810 23:31:47.239689 10644 tablet.cc:2369] T b53d146a30d640e9a56269d54df8c8f7 P 46385dda669f41eca466dc6d65488630: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:31:47.243402 11075 heartbeater.cc:503] Master 127.9.48.254:35559 was elected leader, sending a full tablet report...
I20260810 23:31:47.246248 11101 ts_tablet_manager.cc:1428] T b53d146a30d640e9a56269d54df8c8f7 P 7424e30f6b194f8991181ae6d42ce731: Time spent starting tablet: real 0.026s user 0.016s sys 0.006s
W20260810 23:31:47.294543 10866 tablet.cc:2369] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:31:47.323938 11077 tablet.cc:2369] T b53d146a30d640e9a56269d54df8c8f7 P 7424e30f6b194f8991181ae6d42ce731: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:31:47.370414 11111 raft_consensus.cc:482] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:31:47.370781 11111 raft_consensus.cc:504] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7424e30f6b194f8991181ae6d42ce731" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33543 } } peers { permanent_uuid: "46385dda669f41eca466dc6d65488630" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37985 } } peers { permanent_uuid: "fa05a6aff81945758e6d20e64fa873a1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38743 } }
I20260810 23:31:47.373096 11111 leader_election.cc:288] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7424e30f6b194f8991181ae6d42ce731 (127.9.48.195:33543), 46385dda669f41eca466dc6d65488630 (127.9.48.193:37985)
I20260810 23:31:47.381395 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.373560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:48209 (local address 127.9.48.195:33543)
0810 23:31:47.373866 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:47.373877 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:47.374516 (+ 639us) server_negotiation.cc:407] Connection header received
0810 23:31:47.374875 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:47.374884 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:47.374996 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:47.375077 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:47.375971 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.377183 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.379404 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.380282 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.380436 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.380526 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:47.380977 (+ 451us) server_negotiation.cc:299] Negotiation successful
0810 23:31:47.381113 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:31:47.381418 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.374798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:36167 (local address 127.9.48.193:37985)
0810 23:31:47.374982 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:47.374988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:47.375587 (+ 599us) server_negotiation.cc:407] Connection header received
0810 23:31:47.375759 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:47.375766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:47.375849 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:47.375942 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:47.376474 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.377365 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.379550 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.380323 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.380553 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.380632 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:47.381019 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 23:31:47.381147 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:31:47.381549 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.374561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:37985 (local address 127.9.48.194:36167)
0810 23:31:47.375450 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:47.375467 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:47.375604 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:47.375990 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:47.375996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:47.376021 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:47.376307 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:47.376314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.377509 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.377514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:47.379307 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:47.379340 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.380642 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.380647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:47.380825 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.380848 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:31:47.380963 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 23:31:47.381090 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":19,"thread_start_us":555,"threads_started":1}
I20260810 23:31:47.381798 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.373442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:33543 (local address 127.9.48.194:48209)
0810 23:31:47.374413 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:47.374436 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:47.374638 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:47.375318 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:47.375323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:47.375355 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:47.375807 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:47.375821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.377318 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.377323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:47.379244 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:47.379260 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.380533 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.380560 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:47.380798 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.380828 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:31:47.380905 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 23:31:47.381056 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":90,"thread_start_us":651,"threads_started":1}
I20260810 23:31:47.382872 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b53d146a30d640e9a56269d54df8c8f7" candidate_uuid: "fa05a6aff81945758e6d20e64fa873a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46385dda669f41eca466dc6d65488630" is_pre_election: true
I20260810 23:31:47.382905 11002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b53d146a30d640e9a56269d54df8c8f7" candidate_uuid: "fa05a6aff81945758e6d20e64fa873a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7424e30f6b194f8991181ae6d42ce731" is_pre_election: true
I20260810 23:31:47.383584 10569 raft_consensus.cc:2454] T b53d146a30d640e9a56269d54df8c8f7 P 46385dda669f41eca466dc6d65488630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa05a6aff81945758e6d20e64fa873a1 in term 0.
I20260810 23:31:47.384351 10684 leader_election.cc:302] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46385dda669f41eca466dc6d65488630, fa05a6aff81945758e6d20e64fa873a1; no voters:
I20260810 23:31:47.384793 11111 raft_consensus.cc:2781] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:31:47.384927 11111 raft_consensus.cc:482] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:31:47.385047 11111 raft_consensus.cc:3037] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:47.385596 11002 raft_consensus.cc:2454] T b53d146a30d640e9a56269d54df8c8f7 P 7424e30f6b194f8991181ae6d42ce731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa05a6aff81945758e6d20e64fa873a1 in term 0.
I20260810 23:31:47.387660 11111 raft_consensus.cc:504] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7424e30f6b194f8991181ae6d42ce731" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33543 } } peers { permanent_uuid: "46385dda669f41eca466dc6d65488630" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37985 } } peers { permanent_uuid: "fa05a6aff81945758e6d20e64fa873a1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38743 } }
I20260810 23:31:47.388370 11111 leader_election.cc:288] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [CANDIDATE]: Term 1 election: Requested vote from peers 7424e30f6b194f8991181ae6d42ce731 (127.9.48.195:33543), 46385dda669f41eca466dc6d65488630 (127.9.48.193:37985)
I20260810 23:31:47.388804 11002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b53d146a30d640e9a56269d54df8c8f7" candidate_uuid: "fa05a6aff81945758e6d20e64fa873a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7424e30f6b194f8991181ae6d42ce731"
I20260810 23:31:47.388892 10569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b53d146a30d640e9a56269d54df8c8f7" candidate_uuid: "fa05a6aff81945758e6d20e64fa873a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46385dda669f41eca466dc6d65488630"
I20260810 23:31:47.389017 11002 raft_consensus.cc:3037] T b53d146a30d640e9a56269d54df8c8f7 P 7424e30f6b194f8991181ae6d42ce731 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:47.389142 10569 raft_consensus.cc:3037] T b53d146a30d640e9a56269d54df8c8f7 P 46385dda669f41eca466dc6d65488630 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:47.391338 11002 raft_consensus.cc:2454] T b53d146a30d640e9a56269d54df8c8f7 P 7424e30f6b194f8991181ae6d42ce731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa05a6aff81945758e6d20e64fa873a1 in term 1.
I20260810 23:31:47.391390 10569 raft_consensus.cc:2454] T b53d146a30d640e9a56269d54df8c8f7 P 46385dda669f41eca466dc6d65488630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa05a6aff81945758e6d20e64fa873a1 in term 1.
I20260810 23:31:47.391762 10684 leader_election.cc:302] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46385dda669f41eca466dc6d65488630, fa05a6aff81945758e6d20e64fa873a1; no voters:
I20260810 23:31:47.392076 11111 raft_consensus.cc:2781] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:47.392601 11111 raft_consensus.cc:686] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [term 1 LEADER]: Becoming Leader. State: Replica: fa05a6aff81945758e6d20e64fa873a1, State: Running, Role: LEADER
I20260810 23:31:47.393097 11111 consensus_queue.cc:237] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7424e30f6b194f8991181ae6d42ce731" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33543 } } peers { permanent_uuid: "46385dda669f41eca466dc6d65488630" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37985 } } peers { permanent_uuid: "fa05a6aff81945758e6d20e64fa873a1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38743 } }
I20260810 23:31:47.398145 10348 catalog_manager.cc:5495] T b53d146a30d640e9a56269d54df8c8f7 P fa05a6aff81945758e6d20e64fa873a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa05a6aff81945758e6d20e64fa873a1 (127.9.48.194). New cstate: current_term: 1 leader_uuid: "fa05a6aff81945758e6d20e64fa873a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7424e30f6b194f8991181ae6d42ce731" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46385dda669f41eca466dc6d65488630" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa05a6aff81945758e6d20e64fa873a1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38743 } health_report { overall_health: HEALTHY } } }
Aug 10 23:31:47 dist-test-slave-04bb krb5kdc[10282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:31:47.424703 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.409547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.9.48.193:37985)
0810 23:31:47.409750 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:47.409757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:47.410433 (+ 676us) server_negotiation.cc:407] Connection header received
0810 23:31:47.410650 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:47.410656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:47.410766 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:47.410852 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:47.412346 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.413207 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.414438 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.414867 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.414964 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.419291 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:47.419306 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:47.419324 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:47.419359 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:47.422015 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.422612 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.422619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.422621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.422756 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.423104 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.423107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.423109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.423440 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:47.423661 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:47.424082 (+ 421us) server_negotiation.cc:299] Negotiation successful
0810 23:31:47.424273 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 23:31:47 dist-test-slave-04bb krb5kdc[10282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404707, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:31:47 dist-test-slave-04bb krb5kdc[10282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404707, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:31:47.478022 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.463106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33130 (local address 127.9.48.193:37985)
0810 23:31:47.463328 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:47.463337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:47.463733 (+ 396us) server_negotiation.cc:407] Connection header received
0810 23:31:47.463963 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:47.463967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:47.464035 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:47.464145 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:47.465610 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.466411 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.467676 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.468107 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.468249 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.472183 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:47.472189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:47.472193 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:47.472228 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:47.475353 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.475948 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.475952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.475955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.476132 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.476489 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.476493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.476495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.476987 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:47.477324 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:47.477556 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 23:31:47.477738 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 23:31:47 dist-test-slave-04bb krb5kdc[10282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:31:47 dist-test-slave-04bb krb5kdc[10282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:31:47.528927 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.514472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.9.48.193:37985)
0810 23:31:47.514691 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:47.514699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:47.515056 (+ 357us) server_negotiation.cc:407] Connection header received
0810 23:31:47.515278 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:47.515283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:47.515351 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:47.515462 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:47.516935 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.517712 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.519038 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.519469 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.519607 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.523634 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:47.523640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:47.523643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:47.523675 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:47.526229 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.526884 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.526891 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.526894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.527021 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.527406 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.527411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.527415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.527724 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:47.528003 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:47.528310 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 23:31:47.528578 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 23:31:47.531286 10509 tablet_service.cc:2908] Rejecting scan request for tablet b53d146a30d640e9a56269d54df8c8f7: Uninitialized: clean time has not yet been initialized
I20260810 23:31:47.535251 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 10439
I20260810 23:31:47.548256 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.536457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.9.48.193:37985)
0810 23:31:47.536671 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:47.536676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:47.536996 (+ 320us) server_negotiation.cc:407] Connection header received
0810 23:31:47.537135 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:47.537139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:47.537206 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:47.537288 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:47.538356 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.539022 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.540253 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.540749 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.540898 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.543076 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:47.543082 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:47.543085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:47.543113 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:47.545646 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.546323 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.546327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.546330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.546446 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.546824 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.546831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.546836 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.547103 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:47.547418 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:47.547712 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 23:31:47.547950 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260810 23:31:47.548991 10633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:47.642381 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 10439
I20260810 23:31:47.670495 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 10655
Aug 10 23:31:47 dist-test-slave-04bb krb5kdc[10282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:31:47.686582 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.671576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.9.48.194:38743)
0810 23:31:47.671762 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:47.671767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:47.671784 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:31:47.671850 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:47.671853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:47.671916 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:47.672001 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:47.673403 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.674106 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.675443 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.675840 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.675929 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.680894 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:47.680910 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:47.680927 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:47.680974 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:47.683832 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.684591 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.684599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.684601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.684756 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.685189 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.685192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.685194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.685535 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:47.685762 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:47.686044 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 23:31:47.686221 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:31:47.687280 10858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:47.780814 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 10655
W20260810 23:31:47.805510 10898 connection.cc:535] server connection from 127.9.48.194:48209 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:31:47.805923 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 10876
Aug 10 23:31:47 dist-test-slave-04bb krb5kdc[10282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:31:47.823000 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:47.806834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.9.48.195:33543)
0810 23:31:47.807048 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:47.807057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:47.807082 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:31:47.807350 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:47.807355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:47.807429 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:47.807548 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:47.809272 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.810206 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.811694 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:47.812154 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:47.812314 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:47.816975 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:47.816989 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:47.817005 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:47.817041 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:47.819929 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.820745 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.820753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.820756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.820908 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:47.821346 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:47.821351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:47.821353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:47.821725 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:47.822053 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:47.822377 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 23:31:47.822624 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:31:47.823674 11068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:47.911762 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 10876
I20260810 23:31:47.944962 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 10302
I20260810 23:31:47.946261 10395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:48.056300 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 10302
2026-08-10T23:31:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4672 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:31:48 dist-test-slave-04bb krb5kdc[11149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:31:48 dist-test-slave-04bb krb5kdc[11149](info): set up 2 sockets
Aug 10 23:31:48 dist-test-slave-04bb krb5kdc[11149](info): commencing operation
krb5kdc: starting...
W20260810 23:31:49.433979 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.326s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:31:49 dist-test-slave-04bb krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:31:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:31:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:31:49.640573 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:38367
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:49.977375 11169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:49.977665 11169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:49.977835 11169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:49.985219 11169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:49.985314 11169 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:31:49.985385 11169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:49.985493 11169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:49.985576 11169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:49.985679 11169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:49.995503 11169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:38367
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:49.996919 11169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:49.998292 11169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:50.010600 11177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:50.010699 11179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:50.010689 11181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:50.011955 11169 server_base.cc:1034] running on GCE node
I20260810 23:31:50.012786 11169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:50.014190 11169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:50.015411 11169 hybrid_clock.cc:648] HybridClock initialized: now 1786404710015335 us; error 51 us; skew 500 ppm
I20260810 23:31:50.015730 11169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:50 dist-test-slave-04bb krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404710, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:50.018959 11169 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:31:50.020609 11169 webserver.cc:460] Webserver started at http://127.9.48.254:39761/ using document root <none> and password file <none>
I20260810 23:31:50.021314 11169 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:50.021437 11169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:50.021799 11169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:50.024883 11169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "857cfaa89e6e4276a699be0156c7cbe3"
format_stamp: "Formatted at 2026-08-10 23:31:50 on dist-test-slave-04bb"
server_key: "8dffc86533029af3ae57ef8e99e70878"
server_key_iv: "6301aa8ba68ac7db63abdcfdc35a203b"
server_key_version: "encryptionkey@0"
I20260810 23:31:50.025695 11169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "857cfaa89e6e4276a699be0156c7cbe3"
format_stamp: "Formatted at 2026-08-10 23:31:50 on dist-test-slave-04bb"
server_key: "8dffc86533029af3ae57ef8e99e70878"
server_key_iv: "6301aa8ba68ac7db63abdcfdc35a203b"
server_key_version: "encryptionkey@0"
I20260810 23:31:50.030851 11169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:31:50.034231 11189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:50.035542 11169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:31:50.035722 11169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "857cfaa89e6e4276a699be0156c7cbe3"
format_stamp: "Formatted at 2026-08-10 23:31:50 on dist-test-slave-04bb"
server_key: "8dffc86533029af3ae57ef8e99e70878"
server_key_iv: "6301aa8ba68ac7db63abdcfdc35a203b"
server_key_version: "encryptionkey@0"
I20260810 23:31:50.035919 11169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:50.062824 11169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:50.067178 11169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:50.067489 11169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:50.091185 11169 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:38367
I20260810 23:31:50.091193 11283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:38367 every 8 connection(s)
I20260810 23:31:50.093315 11169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:31:50.098681 11286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:50.101440 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 11169
I20260810 23:31:50.101672 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:31:50.102362 9411 external_mini_cluster.cc:1428] Setting key a7d5e24f1928b0d9847dc5a4b3cd2252
I20260810 23:31:50.111299 11286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3: Bootstrap starting.
I20260810 23:31:50.115198 11286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:50.116601 11286 log.cc:824] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:31:50 dist-test-slave-04bb krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:50.119643 11286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3: No bootstrap required, opened a new log
I20260810 23:31:50.126129 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:50.104563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.9.48.254:38367)
0810 23:31:50.105399 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:50.105411 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:50.105451 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:31:50.107154 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:50.107177 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:50.107566 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:50.107914 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:50.109725 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:50.111001 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:50.112738 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:50.113406 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:50.113550 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:50.118640 (+ 5090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:50.118649 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:50.118663 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:50.118699 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:50.121960 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:50.122707 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:50.122715 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:50.122721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:50.122902 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:50.123334 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:50.123338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:50.123340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:50.123851 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:50.124187 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:50.124669 (+ 482us) server_negotiation.cc:299] Negotiation successful
0810 23:31:50.125099 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":387,"threads_started":1}
I20260810 23:31:50.127970 11286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857cfaa89e6e4276a699be0156c7cbe3" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38367 } }
I20260810 23:31:50.128352 11286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:50.128481 11286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857cfaa89e6e4276a699be0156c7cbe3, State: Initialized, Role: FOLLOWER
I20260810 23:31:50.129158 11286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857cfaa89e6e4276a699be0156c7cbe3" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38367 } }
I20260810 23:31:50.129407 11286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:31:50.129556 11286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:31:50.129743 11286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:50.131922 11286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857cfaa89e6e4276a699be0156c7cbe3" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38367 } }
I20260810 23:31:50.132411 11286 leader_election.cc:302] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 857cfaa89e6e4276a699be0156c7cbe3; no voters:
I20260810 23:31:50.132992 11286 leader_election.cc:288] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:31:50.133136 11296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:50.133877 11296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [term 1 LEADER]: Becoming Leader. State: Replica: 857cfaa89e6e4276a699be0156c7cbe3, State: Running, Role: LEADER
I20260810 23:31:50.134474 11296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857cfaa89e6e4276a699be0156c7cbe3" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38367 } }
I20260810 23:31:50.135049 11286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:50.137379 11297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "857cfaa89e6e4276a699be0156c7cbe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857cfaa89e6e4276a699be0156c7cbe3" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38367 } } }
I20260810 23:31:50.137725 11297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:50.137991 11299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 857cfaa89e6e4276a699be0156c7cbe3. Latest consensus state: current_term: 1 leader_uuid: "857cfaa89e6e4276a699be0156c7cbe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857cfaa89e6e4276a699be0156c7cbe3" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38367 } } }
I20260810 23:31:50.138329 11299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:50.140750 11302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:31:50.146175 11302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:31:50.155005 11302 catalog_manager.cc:1347] Generated new cluster ID: 6b68424710cd41d99786e7c3b0d92c54
I20260810 23:31:50.155124 11302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:31:50.164515 11302 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:31:50.165762 11302 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:31:50.175299 11302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3: Generated new TSK 0
I20260810 23:31:50.176054 11302 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:31:50.232174 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:38367
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:50.581845 11327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:50.582121 11327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:50.582254 11327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:50.589497 11327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:50.589592 11327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:50.589675 11327 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:31:50.589756 11327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:31:50.599220 11327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.48.254:38367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:50.600433 11327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:50.601799 11327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:50.615072 11333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:50.615198 11332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:50.615167 11335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:50.615936 11327 server_base.cc:1034] running on GCE node
I20260810 23:31:50.616613 11327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:50.617566 11327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:50.619268 11327 hybrid_clock.cc:648] HybridClock initialized: now 1786404710619144 us; error 58 us; skew 500 ppm
I20260810 23:31:50.619653 11327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:50 dist-test-slave-04bb krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404710, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:50.623208 11327 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:31:50.624819 11327 webserver.cc:460] Webserver started at http://127.9.48.193:33367/ using document root <none> and password file <none>
I20260810 23:31:50.625552 11327 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:50.625685 11327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:50.626009 11327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:50.629133 11327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "f435bcb4baec4dda9569f0a3e26ca97b"
format_stamp: "Formatted at 2026-08-10 23:31:50 on dist-test-slave-04bb"
server_key: "5ff51837580a75d9aba85483116e21ea"
server_key_iv: "dd8675b43e703038bebdcd82bb3270b8"
server_key_version: "encryptionkey@0"
I20260810 23:31:50.629954 11327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "f435bcb4baec4dda9569f0a3e26ca97b"
format_stamp: "Formatted at 2026-08-10 23:31:50 on dist-test-slave-04bb"
server_key: "5ff51837580a75d9aba85483116e21ea"
server_key_iv: "dd8675b43e703038bebdcd82bb3270b8"
server_key_version: "encryptionkey@0"
I20260810 23:31:50.635040 11327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:31:50.638839 11342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:50.640439 11327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:31:50.640641 11327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "f435bcb4baec4dda9569f0a3e26ca97b"
format_stamp: "Formatted at 2026-08-10 23:31:50 on dist-test-slave-04bb"
server_key: "5ff51837580a75d9aba85483116e21ea"
server_key_iv: "dd8675b43e703038bebdcd82bb3270b8"
server_key_version: "encryptionkey@0"
I20260810 23:31:50.640861 11327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:50.666965 11327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:50.671219 11327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:50.671516 11327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:50.674352 11327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:31:50.674464 11327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:50.674643 11327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:31:50.674726 11327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:31:50 dist-test-slave-04bb krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404710, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:50.701095 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:50.675355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:40359 (local address 127.9.48.254:38367)
0810 23:31:50.675930 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:50.675937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:50.677469 (+ 1532us) server_negotiation.cc:407] Connection header received
0810 23:31:50.680988 (+ 3519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:50.680996 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:50.681100 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:50.681236 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:50.683611 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:50.684971 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:50.687862 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:50.688469 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:50.688698 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:50.694568 (+ 5870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:50.694574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:50.694578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:50.694613 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:50.697537 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:50.698325 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:50.698335 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:50.698339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:50.698477 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:50.699028 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:50.699037 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:50.699044 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:50.699405 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:50.699590 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:50.700349 (+ 759us) server_negotiation.cc:299] Negotiation successful
0810 23:31:50.700621 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":332,"threads_started":1}
I20260810 23:31:50.702147 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:50.675993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:38367 (local address 127.9.48.193:40359)
0810 23:31:50.677283 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:50.677340 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:50.680571 (+ 3231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:50.681540 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:50.681557 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:50.682442 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:50.683371 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:50.683397 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:50.685208 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:50.685216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:50.687577 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:50.687591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:50.688833 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:50.688843 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:50.689207 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:50.690125 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:50.690163 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:50.694307 (+ 4144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:50.697697 (+ 3390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:50.697709 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:50.697725 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:50.698191 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:50.698611 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:50.698617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:50.698621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:50.698828 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:50.699590 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:50.699600 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:50.699953 (+ 353us) client_negotiation.cc:769] Sending connection context
0810 23:31:50.700250 (+ 297us) client_negotiation.cc:240] Negotiation successful
0810 23:31:50.700852 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"thread_start_us":426,"threads_started":1}
I20260810 23:31:50.727663 11327 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:36483
I20260810 23:31:50.727723 11526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:36483 every 8 connection(s)
I20260810 23:31:50.729112 11327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:31:50.729918 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:50.711536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:32975 (local address 127.9.48.254:38367)
0810 23:31:50.711733 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:50.711740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:50.712714 (+ 974us) server_negotiation.cc:407] Connection header received
0810 23:31:50.713034 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:50.713042 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:50.713151 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:50.713263 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:50.715161 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:50.716346 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:50.718949 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:50.719648 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:50.719791 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:50.723057 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:50.723066 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:50.723071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:50.723110 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:50.726747 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:50.727856 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:50.727862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:50.727865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:50.728016 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:50.728391 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:50.728396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:50.728400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:50.728743 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:50.728901 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:50.729315 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 23:31:50.729535 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:31:50.730036 11462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:50.711554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:38367 (local address 127.9.48.193:32975)
0810 23:31:50.712586 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:50.712614 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:50.712852 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:50.713352 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:50.713366 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:50.714231 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:50.714861 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:50.714880 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:50.716546 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:50.716553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:50.718780 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:50.718803 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:50.719921 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:50.719933 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:50.720250 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:50.721127 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:50.721161 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:50.722835 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:50.726953 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:50.726958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:50.726964 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:50.727659 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:50.728130 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:50.728135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:50.728138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:50.728274 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:50.728861 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:50.728867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:50.729051 (+ 184us) client_negotiation.cc:769] Sending connection context
0810 23:31:50.729200 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 23:31:50.729529 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":400,"threads_started":1}
I20260810 23:31:50.733306 11527 heartbeater.cc:347] Connected to a master server at 127.9.48.254:38367
I20260810 23:31:50.733947 11527 heartbeater.cc:464] Registering TS with master...
I20260810 23:31:50.734659 11527 heartbeater.cc:511] Master 127.9.48.254:38367 requested a full tablet report, sending...
I20260810 23:31:50.735086 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 11327
I20260810 23:31:50.735314 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:31:50.736116 9411 external_mini_cluster.cc:1428] Setting key 75df321d72205ff381827ea93b440bc0
I20260810 23:31:50.737066 11217 ts_manager.cc:194] Registered new tserver with Master: f435bcb4baec4dda9569f0a3e26ca97b (127.9.48.193:36483)
I20260810 23:31:50.738735 11217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:40359
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:31:50.781534 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:38367
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:51.124655 11538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:51.124963 11538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:51.125157 11538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:51.132385 11538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:51.132488 11538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:51.132673 11538 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:31:51.132807 11538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:31:51.142925 11538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.48.254:38367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:51.144275 11538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:51.145782 11538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:51.159004 11548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:51.159044 11551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:51.159085 11556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:51.159986 11538 server_base.cc:1034] running on GCE node
I20260810 23:31:51.160635 11538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:51.161533 11538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:51.162750 11538 hybrid_clock.cc:648] HybridClock initialized: now 1786404711162696 us; error 31 us; skew 500 ppm
I20260810 23:31:51.163056 11538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:51 dist-test-slave-04bb krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404711, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:51.166239 11538 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:31:51.167634 11538 webserver.cc:460] Webserver started at http://127.9.48.194:43137/ using document root <none> and password file <none>
I20260810 23:31:51.168355 11538 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:51.168486 11538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:51.168844 11538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:51.171785 11538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "418cb7dc7af147869d4036c9b965a510"
format_stamp: "Formatted at 2026-08-10 23:31:51 on dist-test-slave-04bb"
server_key: "09c66d0b3ee611aad74423a9e29329fa"
server_key_iv: "3a4205c7652b7d5546d15a1273e979bc"
server_key_version: "encryptionkey@0"
I20260810 23:31:51.172644 11538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "418cb7dc7af147869d4036c9b965a510"
format_stamp: "Formatted at 2026-08-10 23:31:51 on dist-test-slave-04bb"
server_key: "09c66d0b3ee611aad74423a9e29329fa"
server_key_iv: "3a4205c7652b7d5546d15a1273e979bc"
server_key_version: "encryptionkey@0"
I20260810 23:31:51.177969 11538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 23:31:51.181360 11565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:51.182597 11538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 23:31:51.182762 11538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "418cb7dc7af147869d4036c9b965a510"
format_stamp: "Formatted at 2026-08-10 23:31:51 on dist-test-slave-04bb"
server_key: "09c66d0b3ee611aad74423a9e29329fa"
server_key_iv: "3a4205c7652b7d5546d15a1273e979bc"
server_key_version: "encryptionkey@0"
I20260810 23:31:51.182974 11538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:51.194832 11538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:51.198676 11538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:51.198952 11538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:51.201661 11538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:31:51.201767 11538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:51.201973 11538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:31:51.202050 11538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:31:51 dist-test-slave-04bb krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404711, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:51.225034 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:51.202074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:33369 (local address 127.9.48.254:38367)
0810 23:31:51.202288 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:51.202294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:51.203666 (+ 1372us) server_negotiation.cc:407] Connection header received
0810 23:31:51.205831 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:51.205837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:51.205980 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:51.206086 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:51.208218 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.209261 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.211692 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.212339 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.212475 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:51.217129 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:51.217138 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:51.217146 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:51.217196 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:51.220903 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:51.221898 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:51.221906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:51.221912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:51.222075 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:51.222703 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:51.222713 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:51.222719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:51.223231 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:51.223444 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:51.224150 (+ 706us) server_negotiation.cc:299] Negotiation successful
0810 23:31:51.224465 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:31:51.225677 11577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:51.202544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:38367 (local address 127.9.48.194:33369)
0810 23:31:51.203480 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:51.203534 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:51.205542 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:51.206322 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:51.206330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:51.207048 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:51.208027 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:51.208063 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.209434 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.209444 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:51.211520 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:51.211538 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.212625 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.212635 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:51.212856 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:51.213721 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:51.213744 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:51.216843 (+ 3099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:51.221043 (+ 4200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:51.221053 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:51.221068 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:51.221711 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:51.222202 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:51.222207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:51.222211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:51.222521 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:51.223392 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:51.223398 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:51.223721 (+ 323us) client_negotiation.cc:769] Sending connection context
0810 23:31:51.223983 (+ 262us) client_negotiation.cc:240] Negotiation successful
0810 23:31:51.224370 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":422,"threads_started":1}
I20260810 23:31:51.249197 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:51.231539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:44365 (local address 127.9.48.254:38367)
0810 23:31:51.231697 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:51.231701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:51.232490 (+ 789us) server_negotiation.cc:407] Connection header received
0810 23:31:51.232709 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:51.232718 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:51.232784 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:51.232881 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:51.234491 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.235967 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.237726 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.238516 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.238683 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:51.241033 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:51.241040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:51.241046 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:51.241097 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:51.245317 (+ 4220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:51.246449 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:51.246453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:51.246456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:51.246675 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:51.247235 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:51.247244 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:51.247250 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:51.247780 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:51.247971 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:51.248498 (+ 527us) server_negotiation.cc:299] Negotiation successful
0810 23:31:51.248787 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:31:51.249398 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:51.231690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:38367 (local address 127.9.48.194:44365)
0810 23:31:51.232392 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:51.232407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:51.232581 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:51.232918 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:51.232926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:51.233839 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:51.234315 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:51.234334 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.236159 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.236167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:51.237561 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:51.237575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.238781 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.238795 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:51.239036 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:51.239772 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:51.239806 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:51.240815 (+ 1009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:51.245461 (+ 4646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:51.245468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:51.245478 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:51.246215 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:51.246844 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:51.246852 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:51.246858 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:51.247064 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:51.247946 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:51.247955 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:51.248214 (+ 259us) client_negotiation.cc:769] Sending connection context
0810 23:31:51.248384 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 23:31:51.248787 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":433,"threads_started":1}
I20260810 23:31:51.260880 11538 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:35279
I20260810 23:31:51.260928 11761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:35279 every 8 connection(s)
I20260810 23:31:51.262073 11538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:31:51.264186 11765 heartbeater.cc:347] Connected to a master server at 127.9.48.254:38367
I20260810 23:31:51.264390 11765 heartbeater.cc:464] Registering TS with master...
I20260810 23:31:51.265105 11765 heartbeater.cc:511] Master 127.9.48.254:38367 requested a full tablet report, sending...
I20260810 23:31:51.266466 11217 ts_manager.cc:194] Registered new tserver with Master: 418cb7dc7af147869d4036c9b965a510 (127.9.48.194:35279)
I20260810 23:31:51.267326 11217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:33369
I20260810 23:31:51.271565 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 11538
I20260810 23:31:51.271749 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:31:51.272149 9411 external_mini_cluster.cc:1428] Setting key 23ec472114cc3b80fd6e0983c8b903d0
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:31:51.323323 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:38367
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:39995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:51.652516 11769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:51.652864 11769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:51.653034 11769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:51.660080 11769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:51.660220 11769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:51.660377 11769 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:31:51.660517 11769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:31:51.669868 11769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:39995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.48.254:38367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:51.671103 11769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:51.672477 11769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:51.686002 11779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:51.686095 11782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:51.685886 11780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:51.687002 11769 server_base.cc:1034] running on GCE node
I20260810 23:31:51.687611 11769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:51.688601 11769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:51.689844 11769 hybrid_clock.cc:648] HybridClock initialized: now 1786404711689780 us; error 37 us; skew 500 ppm
I20260810 23:31:51.690191 11769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:51 dist-test-slave-04bb krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404711, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:51.693418 11769 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:31:51.694909 11769 webserver.cc:460] Webserver started at http://127.9.48.195:33849/ using document root <none> and password file <none>
I20260810 23:31:51.695605 11769 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:51.695735 11769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:51.696069 11769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:51.698999 11769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "b565ee2e3e784ca18fea8dec8771d4dd"
format_stamp: "Formatted at 2026-08-10 23:31:51 on dist-test-slave-04bb"
server_key: "54465c99d648af092285dc72aa990955"
server_key_iv: "30b16789d1cf31a616c400035265bde9"
server_key_version: "encryptionkey@0"
I20260810 23:31:51.699785 11769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "b565ee2e3e784ca18fea8dec8771d4dd"
format_stamp: "Formatted at 2026-08-10 23:31:51 on dist-test-slave-04bb"
server_key: "54465c99d648af092285dc72aa990955"
server_key_iv: "30b16789d1cf31a616c400035265bde9"
server_key_version: "encryptionkey@0"
I20260810 23:31:51.704977 11769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260810 23:31:51.708503 11795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:51.709918 11769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:31:51.710086 11769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "b565ee2e3e784ca18fea8dec8771d4dd"
format_stamp: "Formatted at 2026-08-10 23:31:51 on dist-test-slave-04bb"
server_key: "54465c99d648af092285dc72aa990955"
server_key_iv: "30b16789d1cf31a616c400035265bde9"
server_key_version: "encryptionkey@0"
I20260810 23:31:51.710294 11769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:51.741793 11527 heartbeater.cc:503] Master 127.9.48.254:38367 was elected leader, sending a full tablet report...
I20260810 23:31:51.758430 11769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:51.762908 11769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:51.763150 11769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:51.765956 11769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:31:51.766037 11769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:51.766218 11769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:31:51.766274 11769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:31:51 dist-test-slave-04bb krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404711, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:51.791028 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:51.766632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:46913 (local address 127.9.48.254:38367)
0810 23:31:51.767297 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:51.767304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:51.768502 (+ 1198us) server_negotiation.cc:407] Connection header received
0810 23:31:51.770561 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:51.770571 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:51.770670 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:51.770810 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:51.773045 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.774430 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.777003 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.777411 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.777549 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:51.782996 (+ 5447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:51.783005 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:51.783009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:51.783052 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:51.786367 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:51.787417 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:51.787426 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:51.787429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:51.787602 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:51.788322 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:51.788330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:51.788335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:51.788746 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:51.788914 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:51.790526 (+ 1612us) server_negotiation.cc:299] Negotiation successful
0810 23:31:51.790755 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":400,"threads_started":1}
I20260810 23:31:51.792666 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:51.767349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:38367 (local address 127.9.48.195:46913)
0810 23:31:51.768280 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:51.768359 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:51.770271 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:51.771059 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:51.771067 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:51.771830 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:51.772750 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:51.772775 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.774660 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.774671 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:51.776823 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:51.776836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.777665 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.777674 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:51.777869 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:51.778608 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:51.778646 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:51.782694 (+ 4048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:51.786554 (+ 3860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:51.786566 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:51.786584 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:51.787265 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:51.787762 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:51.787770 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:51.787777 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:51.788157 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:51.788927 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:51.788939 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:51.789538 (+ 599us) client_negotiation.cc:769] Sending connection context
0810 23:31:51.789951 (+ 413us) client_negotiation.cc:240] Negotiation successful
0810 23:31:51.790589 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":425,"threads_started":1}
I20260810 23:31:51.815249 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:51.798602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:38367 (local address 127.9.48.195:34941)
0810 23:31:51.799300 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:51.799322 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:51.799424 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:51.799788 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:51.799791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:51.800205 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:51.800475 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:51.800483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.802248 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.802258 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:51.804050 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:51.804071 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.805377 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.805386 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:51.805597 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:51.806444 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:51.806485 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:51.807846 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:51.811810 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:51.811819 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:51.811825 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:51.812596 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:51.813155 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:51.813163 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:51.813169 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:51.813373 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:51.814203 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:51.814211 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:51.814459 (+ 248us) client_negotiation.cc:769] Sending connection context
0810 23:31:51.814672 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 23:31:51.814910 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":407,"threads_started":1}
I20260810 23:31:51.815956 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:51.798606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:34941 (local address 127.9.48.254:38367)
0810 23:31:51.798854 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:51.798862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:51.799393 (+ 531us) server_negotiation.cc:407] Connection header received
0810 23:31:51.799532 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:51.799540 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:51.799638 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:51.799775 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:51.800650 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.802087 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.804206 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.804893 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.805044 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:51.808072 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:51.808082 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:51.808089 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:51.808133 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:51.811677 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:51.812797 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:51.812806 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:51.812810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:51.813001 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:51.813589 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:51.813597 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:51.813603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:51.814034 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:51.814227 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:51.815562 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0810 23:31:51.815723 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:31:51.820125 11769 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:32877
I20260810 23:31:51.820158 11982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:32877 every 8 connection(s)
I20260810 23:31:51.821434 11769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:31:51.822551 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 11769
I20260810 23:31:51.822757 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:31:51.823324 9411 external_mini_cluster.cc:1428] Setting key 7e6c76b3fc62852308aff65880b3237f
I20260810 23:31:51.823535 11989 heartbeater.cc:347] Connected to a master server at 127.9.48.254:38367
I20260810 23:31:51.823725 11989 heartbeater.cc:464] Registering TS with master...
I20260810 23:31:51.824424 11989 heartbeater.cc:511] Master 127.9.48.254:38367 requested a full tablet report, sending...
I20260810 23:31:51.825911 11216 ts_manager.cc:194] Registered new tserver with Master: b565ee2e3e784ca18fea8dec8771d4dd (127.9.48.195:32877)
I20260810 23:31:51.826813 11216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:46913
I20260810 23:31:51.838724 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:31:51 dist-test-slave-04bb krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:31:51 dist-test-slave-04bb krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:51.891219 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:51.877292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.9.48.254:38367)
0810 23:31:51.877538 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:51.877548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:51.877703 (+ 155us) server_negotiation.cc:407] Connection header received
0810 23:31:51.877873 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:51.877877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:51.877962 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:51.878053 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:51.879445 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.880186 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.881552 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.881965 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.882061 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:51.885881 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:51.885886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:51.885890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:51.885924 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:51.888751 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:51.889352 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:51.889355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:51.889358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:51.889480 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:51.889804 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:51.889808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:51.889811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:51.890072 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:51.890330 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:51.890714 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 23:31:51.890917 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 23:31:51.895793 11216 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:31:51.902523 11216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:31:51.933660 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:51.914919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48494 (local address 127.9.48.193:36483)
0810 23:31:51.916247 (+ 1328us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:51.916256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:51.916285 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:31:51.922450 (+ 6165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:51.922465 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:51.922814 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:51.923061 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:51.924479 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.926604 (+ 2125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.930938 (+ 4334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.932451 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.932642 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:51.932889 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:51.933168 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 23:31:51.933338 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1176,"thread_start_us":1026,"threads_started":1}
I20260810 23:31:51.933709 12010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:51.914675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:36483 (local address 127.0.0.1:48494)
0810 23:31:51.915484 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:51.915532 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:51.915666 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:51.923067 (+ 7401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:51.923074 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:51.923118 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:51.923566 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:51.923581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.926821 (+ 3240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.926826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:51.930743 (+ 3917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:51.930756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.932679 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.932685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:51.932889 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:51.932968 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 23:31:51.933110 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 23:31:51.933319 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":387,"threads_started":1}
I20260810 23:31:51.938089 12014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:51.916118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:35279 (local address 127.0.0.1:33714)
0810 23:31:51.929874 (+ 13756us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:51.929894 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:51.930021 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:51.930989 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:51.930997 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:51.931030 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:51.931372 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:51.931384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.933224 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.933231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:51.935501 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:51.935516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.937323 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.937330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:51.937484 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:51.937517 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:31:51.937606 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:31:51.937737 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13609,"thread_start_us":13228,"threads_started":1}
I20260810 23:31:51.938126 12016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:51.916567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.9.48.194:35279)
0810 23:31:51.928918 (+ 12351us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:51.928931 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:51.930366 (+ 1435us) server_negotiation.cc:407] Connection header received
0810 23:31:51.930471 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:51.930477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:51.930773 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:51.930996 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:51.931564 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.933038 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.935696 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.937128 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.937274 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:51.937497 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:51.937678 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 23:31:51.937842 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12125,"thread_start_us":12375,"threads_started":1}
I20260810 23:31:51.938014 11419 tablet_service.cc:1467] Processing CreateTablet for tablet 479875685e1c4e1b86bf34f5096df95c (DEFAULT_TABLE table=test-table [id=e6c3a12701df417d82790db484f46644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:51.939666 11419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479875685e1c4e1b86bf34f5096df95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:51.940026 12015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:51.916569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:32877 (local address 127.0.0.1:43722)
0810 23:31:51.930436 (+ 13867us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:51.930453 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:51.930568 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:51.931435 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:51.931438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:51.931459 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:51.931915 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:51.931928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.934786 (+ 2858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.934795 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:51.936893 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:51.936910 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.939214 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.939222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:51.939393 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:51.939425 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:31:51.939526 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 23:31:51.939651 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13757,"mutex_wait_us":1,"thread_start_us":14107,"threads_started":1}
I20260810 23:31:51.940235 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:51.916915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.9.48.195:32877)
0810 23:31:51.930007 (+ 13092us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:51.930015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:51.930564 (+ 549us) server_negotiation.cc:407] Connection header received
0810 23:31:51.930735 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:51.930742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:51.930974 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:51.931154 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:51.932247 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.934591 (+ 2344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.937086 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:51.938685 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:51.938839 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:51.939061 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:51.939606 (+ 545us) server_negotiation.cc:299] Negotiation successful
0810 23:31:51.939831 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12904,"thread_start_us":12695,"threads_started":1}
I20260810 23:31:51.941713 11647 tablet_service.cc:1467] Processing CreateTablet for tablet 479875685e1c4e1b86bf34f5096df95c (DEFAULT_TABLE table=test-table [id=e6c3a12701df417d82790db484f46644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:51.943113 11647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479875685e1c4e1b86bf34f5096df95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:51.943733 11869 tablet_service.cc:1467] Processing CreateTablet for tablet 479875685e1c4e1b86bf34f5096df95c (DEFAULT_TABLE table=test-table [id=e6c3a12701df417d82790db484f46644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:51.945204 11869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479875685e1c4e1b86bf34f5096df95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:51.953214 12018 tablet_bootstrap.cc:492] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b: Bootstrap starting.
I20260810 23:31:51.956435 12018 tablet_bootstrap.cc:654] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:51.957574 12018 log.cc:824] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:51.958032 12020 tablet_bootstrap.cc:492] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510: Bootstrap starting.
I20260810 23:31:51.959748 12022 tablet_bootstrap.cc:492] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd: Bootstrap starting.
I20260810 23:31:51.962611 12022 tablet_bootstrap.cc:654] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:51.962625 12020 tablet_bootstrap.cc:654] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:51.963598 12022 log.cc:824] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:51.964156 12020 log.cc:824] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:51.966323 12022 tablet_bootstrap.cc:492] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd: No bootstrap required, opened a new log
I20260810 23:31:51.966622 12022 ts_tablet_manager.cc:1397] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 23:31:51.966848 12018 tablet_bootstrap.cc:492] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b: No bootstrap required, opened a new log
I20260810 23:31:51.967134 12018 ts_tablet_manager.cc:1397] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260810 23:31:51.968837 12020 tablet_bootstrap.cc:492] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510: No bootstrap required, opened a new log
I20260810 23:31:51.969089 12020 ts_tablet_manager.cc:1397] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260810 23:31:51.974625 12022 raft_consensus.cc:348] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:51.975093 12022 raft_consensus.cc:374] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:51.975205 12022 raft_consensus.cc:729] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b565ee2e3e784ca18fea8dec8771d4dd, State: Initialized, Role: FOLLOWER
I20260810 23:31:51.975862 12022 consensus_queue.cc:260] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:51.977033 11989 heartbeater.cc:503] Master 127.9.48.254:38367 was elected leader, sending a full tablet report...
I20260810 23:31:51.977909 12018 raft_consensus.cc:348] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:51.978324 12018 raft_consensus.cc:374] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:51.978412 12018 raft_consensus.cc:729] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f435bcb4baec4dda9569f0a3e26ca97b, State: Initialized, Role: FOLLOWER
I20260810 23:31:51.978988 12022 ts_tablet_manager.cc:1428] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260810 23:31:51.978991 12018 consensus_queue.cc:260] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:51.979806 12020 raft_consensus.cc:348] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:51.980213 12020 raft_consensus.cc:374] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:51.980329 12020 raft_consensus.cc:729] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418cb7dc7af147869d4036c9b965a510, State: Initialized, Role: FOLLOWER
I20260810 23:31:51.980799 12018 ts_tablet_manager.cc:1428] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260810 23:31:51.981005 12020 consensus_queue.cc:260] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:51.982308 11765 heartbeater.cc:503] Master 127.9.48.254:38367 was elected leader, sending a full tablet report...
W20260810 23:31:51.984035 11530 tablet.cc:2369] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:31:51.984274 12020 ts_tablet_manager.cc:1428] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260810 23:31:52.007860 12032 raft_consensus.cc:482] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:31:52.008270 12032 raft_consensus.cc:504] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.009967 12032 leader_election.cc:288] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f435bcb4baec4dda9569f0a3e26ca97b (127.9.48.193:36483), b565ee2e3e784ca18fea8dec8771d4dd (127.9.48.195:32877)
W20260810 23:31:52.014083 11766 tablet.cc:2369] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:31:52.018011 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.010415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:59661 (local address 127.9.48.193:36483)
0810 23:31:52.010603 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:52.010608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:52.011021 (+ 413us) server_negotiation.cc:407] Connection header received
0810 23:31:52.011292 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:52.011297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:52.011400 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:52.011517 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:52.012310 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.013366 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.016035 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.017099 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.017268 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.017345 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:52.017612 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 23:31:52.017744 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:31:52.018097 12034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.010252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:36483 (local address 127.9.48.194:59661)
0810 23:31:52.010897 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:52.010919 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:52.011067 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:52.011710 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:52.011718 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:52.011758 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:52.012155 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.012166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.013488 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.013493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.015879 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.015894 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.017333 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.017338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.017456 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.017481 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:31:52.017566 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:31:52.017654 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":43,"thread_start_us":335,"threads_started":1}
I20260810 23:31:52.018810 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.011367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:49675 (local address 127.9.48.195:32877)
0810 23:31:52.011632 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:52.011642 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:52.012133 (+ 491us) server_negotiation.cc:407] Connection header received
0810 23:31:52.012362 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:52.012366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:52.012425 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:52.012495 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:52.013149 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.014179 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.016490 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.017713 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.017876 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.017942 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:52.018418 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 23:31:52.018549 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:31:52.018920 12036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.011082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:32877 (local address 127.9.48.194:49675)
0810 23:31:52.012033 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:52.012047 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:52.012205 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:52.012566 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:52.012572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:52.012599 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:52.012988 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.013001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.014352 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.014358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.016317 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.016328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.017935 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.017941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.018174 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.018225 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 23:31:52.018391 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 23:31:52.018529 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":4,"thread_start_us":618,"threads_started":1}
I20260810 23:31:52.019243 11455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479875685e1c4e1b86bf34f5096df95c" candidate_uuid: "418cb7dc7af147869d4036c9b965a510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" is_pre_election: true
I20260810 23:31:52.019783 11455 raft_consensus.cc:2454] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418cb7dc7af147869d4036c9b965a510 in term 0.
I20260810 23:31:52.019948 11906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479875685e1c4e1b86bf34f5096df95c" candidate_uuid: "418cb7dc7af147869d4036c9b965a510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" is_pre_election: true
I20260810 23:31:52.020507 11906 raft_consensus.cc:2454] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418cb7dc7af147869d4036c9b965a510 in term 0.
I20260810 23:31:52.020486 11566 leader_election.cc:302] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418cb7dc7af147869d4036c9b965a510, f435bcb4baec4dda9569f0a3e26ca97b; no voters:
I20260810 23:31:52.020941 12032 raft_consensus.cc:2781] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:31:52.021111 12032 raft_consensus.cc:482] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:31:52.021238 12032 raft_consensus.cc:3037] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:52.023501 12032 raft_consensus.cc:504] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.023989 12032 leader_election.cc:288] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [CANDIDATE]: Term 1 election: Requested vote from peers f435bcb4baec4dda9569f0a3e26ca97b (127.9.48.193:36483), b565ee2e3e784ca18fea8dec8771d4dd (127.9.48.195:32877)
I20260810 23:31:52.024653 11455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479875685e1c4e1b86bf34f5096df95c" candidate_uuid: "418cb7dc7af147869d4036c9b965a510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f435bcb4baec4dda9569f0a3e26ca97b"
I20260810 23:31:52.024749 11906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479875685e1c4e1b86bf34f5096df95c" candidate_uuid: "418cb7dc7af147869d4036c9b965a510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b565ee2e3e784ca18fea8dec8771d4dd"
I20260810 23:31:52.024849 11455 raft_consensus.cc:3037] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:52.024981 11906 raft_consensus.cc:3037] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:52.027232 11455 raft_consensus.cc:2454] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418cb7dc7af147869d4036c9b965a510 in term 1.
I20260810 23:31:52.027383 11906 raft_consensus.cc:2454] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418cb7dc7af147869d4036c9b965a510 in term 1.
I20260810 23:31:52.027619 11566 leader_election.cc:302] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418cb7dc7af147869d4036c9b965a510, f435bcb4baec4dda9569f0a3e26ca97b; no voters:
I20260810 23:31:52.027927 12032 raft_consensus.cc:2781] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:52.028412 12032 raft_consensus.cc:686] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [term 1 LEADER]: Becoming Leader. State: Replica: 418cb7dc7af147869d4036c9b965a510, State: Running, Role: LEADER
I20260810 23:31:52.028945 12032 consensus_queue.cc:237] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.033352 11215 catalog_manager.cc:5495] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 418cb7dc7af147869d4036c9b965a510 (127.9.48.194). New cstate: current_term: 1 leader_uuid: "418cb7dc7af147869d4036c9b965a510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260810 23:31:52.068881 12016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.063046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.9.48.194:35279)
0810 23:31:52.063327 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:52.063335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:52.063385 (+ 50us) server_negotiation.cc:407] Connection header received
0810 23:31:52.063714 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:52.063718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:52.063809 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:52.063928 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:52.064274 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.065050 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.066416 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.066832 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.066933 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.067420 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:52.068208 (+ 788us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:52.068332 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:52.068456 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 23:31:52.068571 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260810 23:31:52.073015 11992 tablet.cc:2369] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:31:52.077965 11906 raft_consensus.cc:1261] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd [term 1 FOLLOWER]: Refusing update from remote peer 418cb7dc7af147869d4036c9b965a510: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:31:52.078089 11455 raft_consensus.cc:1261] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b [term 1 FOLLOWER]: Refusing update from remote peer 418cb7dc7af147869d4036c9b965a510: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:31:52.079020 12037 consensus_queue.cc:1035] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:31:52.079401 12032 consensus_queue.cc:1035] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:31:52.090173 12040 mvcc.cc:205] Tried to move back new op lower bound from 7317113700661825536 to 7317113700474404864. Current Snapshot: MvccSnapshot[applied={T|T < 7317113700661825536}]
I20260810 23:31:52.130616 11215 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:31:52.130998 11215 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:31:52.134943 11215 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3: Sending DeleteTablet for 3 replicas of tablet 479875685e1c4e1b86bf34f5096df95c
I20260810 23:31:52.136467 11647 tablet_service.cc:1514] Processing DeleteTablet for tablet 479875685e1c4e1b86bf34f5096df95c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:31:52 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:33714
I20260810 23:31:52.136483 11419 tablet_service.cc:1514] Processing DeleteTablet for tablet 479875685e1c4e1b86bf34f5096df95c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:31:52 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:48494
I20260810 23:31:52.138479 11869 tablet_service.cc:1514] Processing DeleteTablet for tablet 479875685e1c4e1b86bf34f5096df95c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:31:52 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:43722
I20260810 23:31:52.138798 11215 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:31:52.139745 11215 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:31:52.141887 12058 tablet_replica.cc:330] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd: stopping tablet replica
I20260810 23:31:52.142590 12058 raft_consensus.cc:2229] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:31:52.143128 12058 raft_consensus.cc:2258] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:31:52.143970 12057 tablet_replica.cc:330] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b: stopping tablet replica
I20260810 23:31:52.144425 12057 raft_consensus.cc:2229] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:31:52.144956 12057 raft_consensus.cc:2258] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:31:52.147682 12057 ts_tablet_manager.cc:1905] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:31:52.149294 11418 tablet_service.cc:1467] Processing CreateTablet for tablet 09a1c5208934424e84d482eb3e3e8266 (DEFAULT_TABLE table=test-table [id=de76b29fd8224e20a5b9b5ce303ec394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:52.149420 12058 ts_tablet_manager.cc:1905] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:31:52.149820 11418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a1c5208934424e84d482eb3e3e8266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:52.153016 11869 tablet_service.cc:1467] Processing CreateTablet for tablet 09a1c5208934424e84d482eb3e3e8266 (DEFAULT_TABLE table=test-table [id=de76b29fd8224e20a5b9b5ce303ec394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:52.153496 11869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a1c5208934424e84d482eb3e3e8266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:52.153472 11645 tablet_service.cc:1467] Processing CreateTablet for tablet 09a1c5208934424e84d482eb3e3e8266 (DEFAULT_TABLE table=test-table [id=de76b29fd8224e20a5b9b5ce303ec394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:52.154073 11645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a1c5208934424e84d482eb3e3e8266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:52.157450 12018 tablet_bootstrap.cc:492] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b: Bootstrap starting.
I20260810 23:31:52.159150 12018 tablet_bootstrap.cc:654] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:52.160182 12022 tablet_bootstrap.cc:492] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd: Bootstrap starting.
I20260810 23:31:52.161913 12022 tablet_bootstrap.cc:654] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:52.164736 12020 tablet_bootstrap.cc:492] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510: Bootstrap starting.
I20260810 23:31:52.167061 12020 tablet_bootstrap.cc:654] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:52.169131 12022 tablet_bootstrap.cc:492] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd: No bootstrap required, opened a new log
I20260810 23:31:52.169306 12022 ts_tablet_manager.cc:1397] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260810 23:31:52.170444 12022 raft_consensus.cc:348] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.170773 12022 raft_consensus.cc:374] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:52.170894 12022 raft_consensus.cc:729] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b565ee2e3e784ca18fea8dec8771d4dd, State: Initialized, Role: FOLLOWER
I20260810 23:31:52.171252 12022 consensus_queue.cc:260] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.172894 12022 ts_tablet_manager.cc:1428] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:31:52.174407 12055 tablet_replica.cc:330] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510: stopping tablet replica
I20260810 23:31:52.174927 12055 raft_consensus.cc:2229] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:31:52.175855 12055 raft_consensus.cc:2258] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:31:52.177332 12058 ts_tablet_manager.cc:1918] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:31:52.177614 12058 log.cc:1196] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/479875685e1c4e1b86bf34f5096df95c
I20260810 23:31:52.178332 12058 ts_tablet_manager.cc:1939] T 479875685e1c4e1b86bf34f5096df95c P b565ee2e3e784ca18fea8dec8771d4dd: Deleting consensus metadata
I20260810 23:31:52.180246 12055 ts_tablet_manager.cc:1905] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:31:52.180589 11190 catalog_manager.cc:4841] TS b565ee2e3e784ca18fea8dec8771d4dd (127.9.48.195:32877): tablet 479875685e1c4e1b86bf34f5096df95c (table test-table [id=e6c3a12701df417d82790db484f46644]) successfully deleted
I20260810 23:31:52.188582 12055 ts_tablet_manager.cc:1918] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:31:52.188695 12057 ts_tablet_manager.cc:1918] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:31:52.188911 12055 log.cc:1196] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/479875685e1c4e1b86bf34f5096df95c
I20260810 23:31:52.188938 12057 log.cc:1196] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/479875685e1c4e1b86bf34f5096df95c
I20260810 23:31:52.189582 12055 ts_tablet_manager.cc:1939] T 479875685e1c4e1b86bf34f5096df95c P 418cb7dc7af147869d4036c9b965a510: Deleting consensus metadata
I20260810 23:31:52.189671 12057 ts_tablet_manager.cc:1939] T 479875685e1c4e1b86bf34f5096df95c P f435bcb4baec4dda9569f0a3e26ca97b: Deleting consensus metadata
I20260810 23:31:52.191751 11192 catalog_manager.cc:4841] TS 418cb7dc7af147869d4036c9b965a510 (127.9.48.194:35279): tablet 479875685e1c4e1b86bf34f5096df95c (table test-table [id=e6c3a12701df417d82790db484f46644]) successfully deleted
I20260810 23:31:52.193289 12020 tablet_bootstrap.cc:492] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510: No bootstrap required, opened a new log
I20260810 23:31:52.193457 12020 ts_tablet_manager.cc:1397] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.000s
I20260810 23:31:52.193842 12018 tablet_bootstrap.cc:492] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b: No bootstrap required, opened a new log
I20260810 23:31:52.194058 12018 ts_tablet_manager.cc:1397] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.018s
I20260810 23:31:52.194175 11190 catalog_manager.cc:4841] TS f435bcb4baec4dda9569f0a3e26ca97b (127.9.48.193:36483): tablet 479875685e1c4e1b86bf34f5096df95c (table test-table [id=e6c3a12701df417d82790db484f46644]) successfully deleted
I20260810 23:31:52.194458 12020 raft_consensus.cc:348] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.194713 12020 raft_consensus.cc:374] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:52.194846 12020 raft_consensus.cc:729] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418cb7dc7af147869d4036c9b965a510, State: Initialized, Role: FOLLOWER
I20260810 23:31:52.195000 12018 raft_consensus.cc:348] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.195237 12018 raft_consensus.cc:374] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:52.195122 12020 consensus_queue.cc:260] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.195360 12018 raft_consensus.cc:729] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f435bcb4baec4dda9569f0a3e26ca97b, State: Initialized, Role: FOLLOWER
I20260810 23:31:52.195611 12018 consensus_queue.cc:260] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.196393 12018 ts_tablet_manager.cc:1428] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 23:31:52.196899 12020 ts_tablet_manager.cc:1428] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:31:52.411545 12051 raft_consensus.cc:482] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:31:52.411916 12051 raft_consensus.cc:504] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.413045 12051 leader_election.cc:288] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f435bcb4baec4dda9569f0a3e26ca97b (127.9.48.193:36483), b565ee2e3e784ca18fea8dec8771d4dd (127.9.48.195:32877)
I20260810 23:31:52.413688 11906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a1c5208934424e84d482eb3e3e8266" candidate_uuid: "418cb7dc7af147869d4036c9b965a510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" is_pre_election: true
I20260810 23:31:52.413650 11455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a1c5208934424e84d482eb3e3e8266" candidate_uuid: "418cb7dc7af147869d4036c9b965a510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" is_pre_election: true
I20260810 23:31:52.413967 11906 raft_consensus.cc:2454] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418cb7dc7af147869d4036c9b965a510 in term 0.
I20260810 23:31:52.413967 11455 raft_consensus.cc:2454] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418cb7dc7af147869d4036c9b965a510 in term 0.
I20260810 23:31:52.414425 11566 leader_election.cc:302] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418cb7dc7af147869d4036c9b965a510, b565ee2e3e784ca18fea8dec8771d4dd; no voters:
I20260810 23:31:52.414781 12051 raft_consensus.cc:2781] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:31:52.414922 12051 raft_consensus.cc:482] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:31:52.415055 12051 raft_consensus.cc:3037] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:52.417511 12051 raft_consensus.cc:504] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.418026 12051 leader_election.cc:288] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [CANDIDATE]: Term 1 election: Requested vote from peers f435bcb4baec4dda9569f0a3e26ca97b (127.9.48.193:36483), b565ee2e3e784ca18fea8dec8771d4dd (127.9.48.195:32877)
I20260810 23:31:52.418591 11906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a1c5208934424e84d482eb3e3e8266" candidate_uuid: "418cb7dc7af147869d4036c9b965a510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b565ee2e3e784ca18fea8dec8771d4dd"
I20260810 23:31:52.418634 11455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a1c5208934424e84d482eb3e3e8266" candidate_uuid: "418cb7dc7af147869d4036c9b965a510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f435bcb4baec4dda9569f0a3e26ca97b"
I20260810 23:31:52.418795 11906 raft_consensus.cc:3037] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:52.418864 11455 raft_consensus.cc:3037] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:52.420807 11906 raft_consensus.cc:2454] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418cb7dc7af147869d4036c9b965a510 in term 1.
I20260810 23:31:52.421228 11566 leader_election.cc:302] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418cb7dc7af147869d4036c9b965a510, b565ee2e3e784ca18fea8dec8771d4dd; no voters:
I20260810 23:31:52.421502 12051 raft_consensus.cc:2781] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:52.421670 12051 raft_consensus.cc:686] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [term 1 LEADER]: Becoming Leader. State: Replica: 418cb7dc7af147869d4036c9b965a510, State: Running, Role: LEADER
I20260810 23:31:52.421969 12051 consensus_queue.cc:237] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.422272 11455 raft_consensus.cc:2454] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418cb7dc7af147869d4036c9b965a510 in term 1.
I20260810 23:31:52.424930 11215 catalog_manager.cc:5495] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 418cb7dc7af147869d4036c9b965a510 (127.9.48.194). New cstate: current_term: 1 leader_uuid: "418cb7dc7af147869d4036c9b965a510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } health_report { overall_health: UNKNOWN } } }
Aug 10 23:31:52 dist-test-slave-04bb krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404710, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:52.516589 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.495296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:38367 (local address 127.0.0.1:35260)
0810 23:31:52.496095 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:52.496117 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:52.496271 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:52.497072 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:52.497076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:52.497692 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:52.497879 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.497887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.499380 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.499384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.500356 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.500368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.501406 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.501411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.501535 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.507318 (+ 5783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:52.507337 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:52.509854 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:52.513829 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:52.513853 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:52.513858 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:52.514327 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:52.514787 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:52.514790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:52.514793 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:52.514934 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:52.515462 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:52.515472 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:52.515619 (+ 147us) client_negotiation.cc:769] Sending connection context
0810 23:31:52.515801 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 23:31:52.516089 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":438,"threads_started":1}
I20260810 23:31:52.516667 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.495299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.9.48.254:38367)
0810 23:31:52.496747 (+ 1448us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:52.496753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:52.496777 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:31:52.496831 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:52.496836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:52.496913 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:52.497032 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:52.498047 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.499226 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.500514 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.501232 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.501370 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.510077 (+ 8707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:52.510084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:52.510090 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:52.510137 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:52.513683 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:52.514487 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:52.514494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:52.514497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:52.514659 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:52.515044 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:52.515051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:52.515056 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:52.515310 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:52.515453 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:52.515896 (+ 443us) server_negotiation.cc:299] Negotiation successful
0810 23:31:52.516209 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":1313,"thread_start_us":1153,"threads_started":1}
I20260810 23:31:52.522006 11215 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:35260:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:31:52.523234 11215 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:31:52.530227 11418 tablet_service.cc:1467] Processing CreateTablet for tablet 48fca3fa25224886a43e4d178a4437f9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a4d4ee96514444583935a40cb322050]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:31:52.530222 11645 tablet_service.cc:1467] Processing CreateTablet for tablet 48fca3fa25224886a43e4d178a4437f9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a4d4ee96514444583935a40cb322050]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:31:52.530776 11418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48fca3fa25224886a43e4d178a4437f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:52.530790 11645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48fca3fa25224886a43e4d178a4437f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:52.530614 11869 tablet_service.cc:1467] Processing CreateTablet for tablet 48fca3fa25224886a43e4d178a4437f9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a4d4ee96514444583935a40cb322050]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:31:52.531224 11869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48fca3fa25224886a43e4d178a4437f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:52.538611 12020 tablet_bootstrap.cc:492] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510: Bootstrap starting.
I20260810 23:31:52.538640 12022 tablet_bootstrap.cc:492] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd: Bootstrap starting.
I20260810 23:31:52.540212 12020 tablet_bootstrap.cc:654] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:52.540307 12022 tablet_bootstrap.cc:654] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:52.541316 12018 tablet_bootstrap.cc:492] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b: Bootstrap starting.
I20260810 23:31:52.542433 12020 tablet_bootstrap.cc:492] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510: No bootstrap required, opened a new log
I20260810 23:31:52.542555 12020 ts_tablet_manager.cc:1397] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 23:31:52.542888 12018 tablet_bootstrap.cc:654] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:52.543087 12020 raft_consensus.cc:348] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.543291 12020 raft_consensus.cc:374] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:52.543349 12020 raft_consensus.cc:729] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418cb7dc7af147869d4036c9b965a510, State: Initialized, Role: FOLLOWER
I20260810 23:31:52.543628 12020 consensus_queue.cc:260] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.544189 12051 tablet_replica.cc:439] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } } }
I20260810 23:31:52.544476 12051 tablet_replica.cc:442] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:31:52.545511 12020 ts_tablet_manager.cc:1428] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:31:52.549350 12018 tablet_bootstrap.cc:492] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b: No bootstrap required, opened a new log
I20260810 23:31:52.549518 12018 ts_tablet_manager.cc:1397] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260810 23:31:52.550232 12018 raft_consensus.cc:348] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.550465 12018 raft_consensus.cc:374] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:52.550536 12018 raft_consensus.cc:729] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f435bcb4baec4dda9569f0a3e26ca97b, State: Initialized, Role: FOLLOWER
I20260810 23:31:52.550760 12018 consensus_queue.cc:260] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.551301 12018 ts_tablet_manager.cc:1428] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 23:31:52.551272 12031 tablet_replica.cc:439] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } } }
I20260810 23:31:52.551641 12031 tablet_replica.cc:442] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:31:52.555271 12022 tablet_bootstrap.cc:492] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd: No bootstrap required, opened a new log
I20260810 23:31:52.555460 12022 ts_tablet_manager.cc:1397] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20260810 23:31:52.556053 12022 raft_consensus.cc:348] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.556277 12022 raft_consensus.cc:374] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:52.556375 12022 raft_consensus.cc:729] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b565ee2e3e784ca18fea8dec8771d4dd, State: Initialized, Role: FOLLOWER
I20260810 23:31:52.556643 12022 consensus_queue.cc:260] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.557215 12022 ts_tablet_manager.cc:1428] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:31:52.557147 12030 tablet_replica.cc:439] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } } }
I20260810 23:31:52.557507 12030 tablet_replica.cc:442] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:31:52.638286 12030 raft_consensus.cc:482] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:31:52.638496 12030 raft_consensus.cc:504] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.640069 12030 leader_election.cc:288] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f435bcb4baec4dda9569f0a3e26ca97b (127.9.48.193:36483), 418cb7dc7af147869d4036c9b965a510 (127.9.48.194:35279)
I20260810 23:31:52.658515 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.640596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:60997 (local address 127.9.48.194:35279)
0810 23:31:52.652788 (+ 12192us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:52.652796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:52.652832 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:31:52.652967 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:52.652973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:52.653058 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:52.653168 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:52.653751 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.654569 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.656550 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.657480 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.657607 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.657660 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:52.658030 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 23:31:52.658181 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12045,"thread_start_us":11935,"threads_started":1}
I20260810 23:31:52.658535 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.640401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:35279 (local address 127.9.48.195:60997)
0810 23:31:52.652607 (+ 12206us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:52.652640 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:52.652831 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:52.653269 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:52.653278 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:52.653314 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:52.653574 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.653589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.654716 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.654722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.656359 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.656373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.657653 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.657660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.657849 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.657888 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 23:31:52.658003 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 23:31:52.658154 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12029,"mutex_wait_us":1,"thread_start_us":11875,"threads_started":1}
I20260810 23:31:52.659018 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.640639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:51307 (local address 127.9.48.193:36483)
0810 23:31:52.652364 (+ 11725us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:52.652374 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:52.653174 (+ 800us) server_negotiation.cc:407] Connection header received
0810 23:31:52.653357 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:52.653363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:52.653438 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:52.653549 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:52.654096 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.655151 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.657009 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.657979 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.658153 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.658221 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:52.658584 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 23:31:52.658764 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11548,"thread_start_us":11395,"threads_started":1}
I20260810 23:31:52.659044 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.640401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:36483 (local address 127.9.48.195:51307)
0810 23:31:52.653033 (+ 12632us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:52.653052 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:52.653188 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:52.653576 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:52.653579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:52.653600 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:52.653943 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.653955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.655295 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.655299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.656849 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.656864 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.658180 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.658186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.658390 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.658423 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:31:52.658535 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 23:31:52.658689 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12516,"mutex_wait_us":112,"thread_start_us":12225,"threads_started":1}
I20260810 23:31:52.659713 11455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48fca3fa25224886a43e4d178a4437f9" candidate_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" is_pre_election: true
I20260810 23:31:52.659843 11695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48fca3fa25224886a43e4d178a4437f9" candidate_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418cb7dc7af147869d4036c9b965a510" is_pre_election: true
I20260810 23:31:52.660064 11455 raft_consensus.cc:2454] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b565ee2e3e784ca18fea8dec8771d4dd in term 0.
I20260810 23:31:52.660228 11695 raft_consensus.cc:2454] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b565ee2e3e784ca18fea8dec8771d4dd in term 0.
I20260810 23:31:52.660691 11796 leader_election.cc:302] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b565ee2e3e784ca18fea8dec8771d4dd, f435bcb4baec4dda9569f0a3e26ca97b; no voters:
I20260810 23:31:52.661216 12030 raft_consensus.cc:2781] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:31:52.661355 12030 raft_consensus.cc:482] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:31:52.661468 12030 raft_consensus.cc:3037] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:52.663569 12030 raft_consensus.cc:504] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.664172 12030 leader_election.cc:288] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [CANDIDATE]: Term 1 election: Requested vote from peers f435bcb4baec4dda9569f0a3e26ca97b (127.9.48.193:36483), 418cb7dc7af147869d4036c9b965a510 (127.9.48.194:35279)
I20260810 23:31:52.664590 11455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48fca3fa25224886a43e4d178a4437f9" candidate_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f435bcb4baec4dda9569f0a3e26ca97b"
I20260810 23:31:52.664714 11695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48fca3fa25224886a43e4d178a4437f9" candidate_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "418cb7dc7af147869d4036c9b965a510"
I20260810 23:31:52.664846 11455 raft_consensus.cc:3037] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:52.664918 11695 raft_consensus.cc:3037] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:52.667807 11455 raft_consensus.cc:2454] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b565ee2e3e784ca18fea8dec8771d4dd in term 1.
I20260810 23:31:52.667935 11695 raft_consensus.cc:2454] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b565ee2e3e784ca18fea8dec8771d4dd in term 1.
I20260810 23:31:52.668236 11796 leader_election.cc:302] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b565ee2e3e784ca18fea8dec8771d4dd, f435bcb4baec4dda9569f0a3e26ca97b; no voters:
I20260810 23:31:52.668515 12030 raft_consensus.cc:2781] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:52.668943 12030 raft_consensus.cc:686] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [term 1 LEADER]: Becoming Leader. State: Replica: b565ee2e3e784ca18fea8dec8771d4dd, State: Running, Role: LEADER
I20260810 23:31:52.669335 12030 consensus_queue.cc:237] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } }
I20260810 23:31:52.671495 12094 tablet_replica.cc:439] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd: TxnStatusTablet state changed. Reason: New leader b565ee2e3e784ca18fea8dec8771d4dd. Latest consensus state: current_term: 1 leader_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } } }
I20260810 23:31:52.671916 12094 tablet_replica.cc:442] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:31:52.673661 12096 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:31:52.673406 11215 catalog_manager.cc:5495] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd reported cstate change: term changed from 0 to 1, leader changed from <none> to b565ee2e3e784ca18fea8dec8771d4dd (127.9.48.195). New cstate: current_term: 1 leader_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } health_report { overall_health: HEALTHY } } }
I20260810 23:31:52.673977 12096 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:31:52.717394 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.710379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:32877 (local address 127.0.0.1:43724)
0810 23:31:52.710663 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:52.710688 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:52.710845 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:52.711406 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:52.711412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:52.711427 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:31:52.711649 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.711663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.712804 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.712811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.714484 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.714503 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.715293 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.715300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.715529 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.715609 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:31:52.716818 (+ 1209us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:31:52.716841 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:31:52.716936 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 23:31:52.717088 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 23:31:52.717466 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.710381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.9.48.195:32877)
0810 23:31:52.711021 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:52.711026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:52.711042 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:31:52.711118 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:52.711121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:52.711203 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:52.711351 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:52.711828 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.712668 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.714633 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.715120 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.715217 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.715859 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:52.716680 (+ 821us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:52.716845 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:52.716993 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 23:31:52.717079 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":416,"threads_started":1}
I20260810 23:31:52.722142 11455 raft_consensus.cc:1261] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b [term 1 FOLLOWER]: Refusing update from remote peer b565ee2e3e784ca18fea8dec8771d4dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:31:52.722565 11695 raft_consensus.cc:1261] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510 [term 1 FOLLOWER]: Refusing update from remote peer b565ee2e3e784ca18fea8dec8771d4dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:31:52.722885 12030 consensus_queue.cc:1035] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:31:52.723379 12094 consensus_queue.cc:1035] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:31:52.726871 12051 tablet_replica.cc:439] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510: TxnStatusTablet state changed. Reason: New leader b565ee2e3e784ca18fea8dec8771d4dd. Latest consensus state: current_term: 1 leader_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } } }
I20260810 23:31:52.727208 12051 tablet_replica.cc:442] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:31:52.729050 12030 tablet_replica.cc:439] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } } }
I20260810 23:31:52.729411 12030 tablet_replica.cc:442] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:31:52.732097 12106 mvcc.cc:205] Tried to move back new op lower bound from 7317113703302742016 to 7317113703097274368. Current Snapshot: MvccSnapshot[applied={T|T < 7317113703302742016 or (T in {7317113703302742016})}]
I20260810 23:31:52.766351 12051 tablet_replica.cc:439] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } } }
I20260810 23:31:52.766712 12051 tablet_replica.cc:442] T 48fca3fa25224886a43e4d178a4437f9 P 418cb7dc7af147869d4036c9b965a510: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:31:52.766896 12031 tablet_replica.cc:439] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b: TxnStatusTablet state changed. Reason: New leader b565ee2e3e784ca18fea8dec8771d4dd. Latest consensus state: current_term: 1 leader_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } } }
I20260810 23:31:52.767290 12031 tablet_replica.cc:442] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:31:52.769855 12030 tablet_replica.cc:439] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } } }
I20260810 23:31:52.770123 12030 tablet_replica.cc:442] T 48fca3fa25224886a43e4d178a4437f9 P b565ee2e3e784ca18fea8dec8771d4dd: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:31:52.783089 12031 tablet_replica.cc:439] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 } } peers { permanent_uuid: "418cb7dc7af147869d4036c9b965a510" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35279 } } peers { permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 } } }
I20260810 23:31:52.783351 12031 tablet_replica.cc:442] T 48fca3fa25224886a43e4d178a4437f9 P f435bcb4baec4dda9569f0a3e26ca97b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:31:52.793048 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.787347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:45085 (local address 127.9.48.195:32877)
0810 23:31:52.787529 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:52.787538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:52.788059 (+ 521us) server_negotiation.cc:407] Connection header received
0810 23:31:52.788275 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:52.788281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:52.788364 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:52.788460 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:52.788954 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.789735 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.791118 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.791554 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.791699 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.792240 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:52.792346 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:52.792455 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:52.792634 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 23:31:52.792712 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:31:52.793128 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.787235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:32877 (local address 127.9.48.194:45085)
0810 23:31:52.787944 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:52.787962 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:52.788133 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:52.788491 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:52.788495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:52.788510 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:31:52.788803 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.788813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.789900 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.789906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.790974 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.790986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.791734 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.791742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.791874 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.791929 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:31:52.792457 (+ 528us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:31:52.792471 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 23:31:52.792605 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 23:31:52.792746 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":353,"threads_started":1}
I20260810 23:31:52.803009 11906 raft_consensus.cc:1261] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd [term 1 FOLLOWER]: Refusing update from remote peer 418cb7dc7af147869d4036c9b965a510: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:31:52.803072 11455 raft_consensus.cc:1261] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b [term 1 FOLLOWER]: Refusing update from remote peer 418cb7dc7af147869d4036c9b965a510: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:31:52.803601 12051 consensus_queue.cc:1035] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f435bcb4baec4dda9569f0a3e26ca97b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:31:52.803929 12032 consensus_queue.cc:1035] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b565ee2e3e784ca18fea8dec8771d4dd" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:31:52.842834 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.835404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:51865 (local address 127.9.48.194:35279)
0810 23:31:52.835606 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:52.835613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:52.836432 (+ 819us) server_negotiation.cc:407] Connection header received
0810 23:31:52.836765 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:52.836770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:52.836900 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:52.836974 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:52.837884 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.838692 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.840692 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.841280 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.841379 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.842133 (+ 754us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:52.842247 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:52.842357 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:52.842474 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 23:31:52.842552 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:31:52.843118 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.835485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:35279 (local address 127.9.48.195:51865)
0810 23:31:52.836331 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:52.836352 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:52.836591 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:52.837209 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:52.837213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:52.837225 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:31:52.837670 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.837684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.838882 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.838887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.840472 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:52.840543 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.841617 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.841623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:52.841833 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.841910 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:31:52.842359 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:31:52.842373 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 23:31:52.842456 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:31:52.842591 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":1262,"threads_started":1}
I20260810 23:31:52.890789 11216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:31:52.891062 11216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:31:52.894143 11216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 857cfaa89e6e4276a699be0156c7cbe3: Sending DeleteTablet for 3 replicas of tablet 09a1c5208934424e84d482eb3e3e8266
I20260810 23:31:52.895174 11418 tablet_service.cc:1514] Processing DeleteTablet for tablet 09a1c5208934424e84d482eb3e3e8266 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:31:52 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:48494
I20260810 23:31:52.895370 11645 tablet_service.cc:1514] Processing DeleteTablet for tablet 09a1c5208934424e84d482eb3e3e8266 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:31:52 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:33714
I20260810 23:31:52.895638 11869 tablet_service.cc:1514] Processing DeleteTablet for tablet 09a1c5208934424e84d482eb3e3e8266 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:31:52 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:43722
I20260810 23:31:52.906322 12137 tablet_replica.cc:330] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510: stopping tablet replica
I20260810 23:31:52.906548 12137 raft_consensus.cc:2229] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:31:52.907014 12137 raft_consensus.cc:2258] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:31:52.908298 12137 ts_tablet_manager.cc:1905] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:31:52.915997 12137 ts_tablet_manager.cc:1918] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:31:52.916142 12137 log.cc:1196] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/09a1c5208934424e84d482eb3e3e8266
I20260810 23:31:52.917033 12133 tablet_replica.cc:330] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b: stopping tablet replica
I20260810 23:31:52.917268 12133 raft_consensus.cc:2229] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:31:52.917555 12133 raft_consensus.cc:2258] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:31:52.918444 12138 tablet_replica.cc:330] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd: stopping tablet replica
I20260810 23:31:52.918658 12138 raft_consensus.cc:2229] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd [term 1 FOLLOWER]: Raft consensus shutting down.
Aug 10 23:31:52 dist-test-slave-04bb krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:31:52.918745 12133 ts_tablet_manager.cc:1905] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:31:52.918942 12138 raft_consensus.cc:2258] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:31:52.920181 12138 ts_tablet_manager.cc:1905] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:31:52.926440 12133 ts_tablet_manager.cc:1918] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:31:52.926581 12133 log.cc:1196] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/09a1c5208934424e84d482eb3e3e8266
I20260810 23:31:52.927141 12133 ts_tablet_manager.cc:1939] T 09a1c5208934424e84d482eb3e3e8266 P f435bcb4baec4dda9569f0a3e26ca97b: Deleting consensus metadata
I20260810 23:31:52.927628 12138 ts_tablet_manager.cc:1918] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:31:52.927772 12138 log.cc:1196] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/09a1c5208934424e84d482eb3e3e8266
I20260810 23:31:52.928228 12138 ts_tablet_manager.cc:1939] T 09a1c5208934424e84d482eb3e3e8266 P b565ee2e3e784ca18fea8dec8771d4dd: Deleting consensus metadata
I20260810 23:31:52.928402 11190 catalog_manager.cc:4841] TS f435bcb4baec4dda9569f0a3e26ca97b (127.9.48.193:36483): tablet 09a1c5208934424e84d482eb3e3e8266 (table test-table [id=de76b29fd8224e20a5b9b5ce303ec394]) successfully deleted
I20260810 23:31:52.928737 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.906036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48504 (local address 127.9.48.193:36483)
0810 23:31:52.906309 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:52.906319 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:52.907319 (+ 1000us) server_negotiation.cc:407] Connection header received
0810 23:31:52.907605 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:52.907610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:52.907700 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:52.907815 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:52.910088 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.911220 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.913122 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.913777 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.913900 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.920054 (+ 6154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:52.920069 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:52.920095 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:52.920135 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:52.924471 (+ 4336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:52.925449 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:52.925459 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:52.925463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:52.925624 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:52.926063 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:52.926071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:52.926076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:52.926723 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:52.927105 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:52.927983 (+ 878us) server_negotiation.cc:299] Negotiation successful
0810 23:31:52.928239 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:31:52.928918 12137 ts_tablet_manager.cc:1939] T 09a1c5208934424e84d482eb3e3e8266 P 418cb7dc7af147869d4036c9b965a510: Deleting consensus metadata
I20260810 23:31:52.929307 11190 catalog_manager.cc:4841] TS b565ee2e3e784ca18fea8dec8771d4dd (127.9.48.195:32877): tablet 09a1c5208934424e84d482eb3e3e8266 (table test-table [id=de76b29fd8224e20a5b9b5ce303ec394]) successfully deleted
I20260810 23:31:52.930040 11192 catalog_manager.cc:4841] TS 418cb7dc7af147869d4036c9b965a510 (127.9.48.194:35279): tablet 09a1c5208934424e84d482eb3e3e8266 (table test-table [id=de76b29fd8224e20a5b9b5ce303ec394]) successfully deleted
W20260810 23:31:52.930605 11520 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48504
I20260810 23:31:52.947839 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.933759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.9.48.254:38367)
0810 23:31:52.934037 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:52.934045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:52.934259 (+ 214us) server_negotiation.cc:407] Connection header received
0810 23:31:52.934412 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:52.934416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:52.934482 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:52.934559 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:52.936199 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.937015 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.938835 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.939413 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.939513 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.942520 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:52.942525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:52.942528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:52.942561 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:52.945342 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:52.946099 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:52.946103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:52.946105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:52.946220 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:52.946595 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:52.946599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:52.946601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:52.946862 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:52.947046 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:52.947417 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 23:31:52.947575 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
W20260810 23:31:52.948467 11216 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35266
I20260810 23:31:52.950345 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11327
I20260810 23:31:52.963821 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.951556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48518 (local address 127.9.48.193:36483)
0810 23:31:52.951770 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:52.951776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:52.952554 (+ 778us) server_negotiation.cc:407] Connection header received
0810 23:31:52.952747 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:52.952753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:52.952828 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:52.952929 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:52.954003 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.954712 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.955879 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.956298 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.956450 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:52.958635 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:52.958642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:52.958646 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:52.958685 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:52.961291 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:52.961921 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:52.961926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:52.961930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:52.962056 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:52.962418 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:52.962424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:52.962427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:52.962707 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:52.962968 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:52.963260 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 23:31:52.963498 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 23:31:52.964581 11520 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48518
W20260810 23:31:52.965175 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11327: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:31:52.965369 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 11327
I20260810 23:31:52.993844 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11538
Aug 10 23:31:52 dist-test-slave-04bb krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:31:53.009311 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:52.994771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.9.48.194:35279)
0810 23:31:52.995072 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:52.995082 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:52.995103 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:31:52.995176 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:52.995179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:52.995313 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:52.995428 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:52.996594 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.997361 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.998600 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:52.999023 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:52.999157 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:53.003258 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:53.003272 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:53.003287 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:53.003323 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:53.006343 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:53.007070 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:53.007079 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:53.007081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:53.007233 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:53.007625 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:53.007630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:53.007633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:53.008086 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:53.008395 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:53.008709 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 23:31:53.008951 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 23:31:53.010001 11754 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33728
W20260810 23:31:53.010691 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11538: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:31:53.010875 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 11538
I20260810 23:31:53.053966 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11769
Aug 10 23:31:53 dist-test-slave-04bb krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:31:53.072192 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:53.055312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.9.48.195:32877)
0810 23:31:53.055555 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:53.055566 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:53.055598 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:31:53.055672 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:53.055677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:53.055773 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:53.055895 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:53.057231 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:53.058449 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:53.059677 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:53.060342 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:53.060477 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:53.064726 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:53.064749 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:53.064768 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:53.064820 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:53.069197 (+ 4377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:53.069868 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:53.069877 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:53.069880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:53.070042 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:53.070413 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:53.070428 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:53.070431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:53.070957 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:53.071345 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:53.071528 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 23:31:53.071797 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 23:31:53.073165 11977 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43728
W20260810 23:31:53.073911 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11769: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:31:53.074119 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 11769
I20260810 23:31:53.110471 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11169
I20260810 23:31:53.111627 11277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:31:53.228255 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 11169
2026-08-10T23:31:53Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5175 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:31:53 dist-test-slave-04bb krb5kdc[12150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:31:53 dist-test-slave-04bb krb5kdc[12150](info): set up 2 sockets
Aug 10 23:31:53 dist-test-slave-04bb krb5kdc[12150](info): commencing operation
krb5kdc: starting...
W20260810 23:31:54.574499 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:31:54 dist-test-slave-04bb krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:31:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:31:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:31:54.785089 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:43257
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:34179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:43257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:55.123646 12171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:55.123994 12171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:55.124158 12171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:55.131239 12171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:31:55.131377 12171 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:31:55.131471 12171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:55.131582 12171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:55.131675 12171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:31:55.131767 12171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:31:55.141414 12171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:34179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:43257
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:43257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:55.142740 12171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:55.144086 12171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:55.155951 12184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:55.156303 12182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:55.156329 12180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:55.157042 12171 server_base.cc:1034] running on GCE node
I20260810 23:31:55.157796 12171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:55.159107 12171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:55.160342 12171 hybrid_clock.cc:648] HybridClock initialized: now 1786404715160268 us; error 38 us; skew 500 ppm
I20260810 23:31:55.160712 12171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:55 dist-test-slave-04bb krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404715, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:55.163851 12171 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:31:55.165424 12171 webserver.cc:460] Webserver started at http://127.9.48.254:37761/ using document root <none> and password file <none>
I20260810 23:31:55.166112 12171 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:55.166249 12171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:55.166558 12171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:55.169550 12171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "d3df99f497fc4475b2e702612db07d57"
format_stamp: "Formatted at 2026-08-10 23:31:55 on dist-test-slave-04bb"
server_key: "aea432091cc66febfb2a6173b46ba047"
server_key_iv: "64fddd6276834a309c46e6a4d9d98dba"
server_key_version: "encryptionkey@0"
I20260810 23:31:55.170315 12171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "d3df99f497fc4475b2e702612db07d57"
format_stamp: "Formatted at 2026-08-10 23:31:55 on dist-test-slave-04bb"
server_key: "aea432091cc66febfb2a6173b46ba047"
server_key_iv: "64fddd6276834a309c46e6a4d9d98dba"
server_key_version: "encryptionkey@0"
I20260810 23:31:55.175292 12171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:31:55.178738 12196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:55.180042 12171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:31:55.180284 12171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "d3df99f497fc4475b2e702612db07d57"
format_stamp: "Formatted at 2026-08-10 23:31:55 on dist-test-slave-04bb"
server_key: "aea432091cc66febfb2a6173b46ba047"
server_key_iv: "64fddd6276834a309c46e6a4d9d98dba"
server_key_version: "encryptionkey@0"
I20260810 23:31:55.180518 12171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:55.191439 12171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:55.195307 12171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:55.195569 12171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:55.217837 12171 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:43257
I20260810 23:31:55.217881 12293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:43257 every 8 connection(s)
I20260810 23:31:55.219257 12171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:31:55.223079 12294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:55.224742 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 12171
I20260810 23:31:55.224995 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:31:55.225868 9411 external_mini_cluster.cc:1428] Setting key 848e182336ec45c1d1004b599e418a6d
I20260810 23:31:55.237811 12294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57: Bootstrap starting.
Aug 10 23:31:55 dist-test-slave-04bb krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:55.241825 12294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:55.242969 12294 log.cc:824] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:55.246110 12294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57: No bootstrap required, opened a new log
I20260810 23:31:55.250327 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:55.228045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.9.48.254:43257)
0810 23:31:55.228931 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:55.228943 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:55.228995 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:31:55.230631 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:55.230654 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:55.231051 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:55.231381 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:55.233251 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:55.234614 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:55.236449 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:55.237176 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:55.237315 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:55.242170 (+ 4855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:55.242178 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:55.242196 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:55.242240 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:55.245870 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:55.246689 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:55.246696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:55.246703 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:55.246881 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:55.247311 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:55.247317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:55.247324 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:55.247923 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:55.248279 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:55.248828 (+ 549us) server_negotiation.cc:299] Negotiation successful
0810 23:31:55.249258 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":420,"threads_started":1}
I20260810 23:31:55.254037 12294 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3df99f497fc4475b2e702612db07d57" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43257 } }
I20260810 23:31:55.254395 12294 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:55.254506 12294 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3df99f497fc4475b2e702612db07d57, State: Initialized, Role: FOLLOWER
I20260810 23:31:55.255026 12294 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3df99f497fc4475b2e702612db07d57" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43257 } }
I20260810 23:31:55.255271 12294 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:31:55.255415 12294 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:31:55.255587 12294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:55.257829 12294 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3df99f497fc4475b2e702612db07d57" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43257 } }
I20260810 23:31:55.258351 12294 leader_election.cc:302] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3df99f497fc4475b2e702612db07d57; no voters:
I20260810 23:31:55.259029 12294 leader_election.cc:288] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:31:55.259150 12305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:55.259886 12305 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [term 1 LEADER]: Becoming Leader. State: Replica: d3df99f497fc4475b2e702612db07d57, State: Running, Role: LEADER
I20260810 23:31:55.260417 12305 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3df99f497fc4475b2e702612db07d57" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43257 } }
I20260810 23:31:55.261480 12294 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:31:55.263845 12308 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3df99f497fc4475b2e702612db07d57. Latest consensus state: current_term: 1 leader_uuid: "d3df99f497fc4475b2e702612db07d57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3df99f497fc4475b2e702612db07d57" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43257 } } }
I20260810 23:31:55.264219 12308 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:55.266685 12319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:31:55.266548 12306 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3df99f497fc4475b2e702612db07d57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3df99f497fc4475b2e702612db07d57" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43257 } } }
I20260810 23:31:55.266865 12306 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57 [sys.catalog]: This master's current role is: LEADER
I20260810 23:31:55.271335 12319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:31:55.279357 12319 catalog_manager.cc:1347] Generated new cluster ID: cf43b1fad8c74b9893812ba176f872f9
I20260810 23:31:55.279464 12319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:31:55.287061 12319 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:31:55.288852 12319 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:31:55.305254 12319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57: Generated new TSK 0
I20260810 23:31:55.306140 12319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:31:55.369624 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:43257
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:34179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:55.699113 12347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:55.699402 12347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:55.699604 12347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:55.706693 12347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:55.706789 12347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:55.706872 12347 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:31:55.706986 12347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:31:55.707095 12347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:31:55.716456 12347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:34179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.48.254:43257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:55.717701 12347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:55.718919 12347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:55.731956 12363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:55.731913 12357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:55.731994 12360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:55.733012 12347 server_base.cc:1034] running on GCE node
I20260810 23:31:55.733594 12347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:55.734470 12347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:55.735698 12347 hybrid_clock.cc:648] HybridClock initialized: now 1786404715735654 us; error 53 us; skew 500 ppm
I20260810 23:31:55.736004 12347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:55 dist-test-slave-04bb krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404715, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:55.739048 12347 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:31:55.740466 12347 webserver.cc:460] Webserver started at http://127.9.48.193:34005/ using document root <none> and password file <none>
I20260810 23:31:55.741191 12347 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:55.741331 12347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:55.741636 12347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:55.744395 12347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "45affa7eaf354e259778257c2d75662a"
format_stamp: "Formatted at 2026-08-10 23:31:55 on dist-test-slave-04bb"
server_key: "cee98424579bd3ea8b85e5c004180636"
server_key_iv: "6600ed8fcad408d2e78b025b1b6b4af0"
server_key_version: "encryptionkey@0"
I20260810 23:31:55.745195 12347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "45affa7eaf354e259778257c2d75662a"
format_stamp: "Formatted at 2026-08-10 23:31:55 on dist-test-slave-04bb"
server_key: "cee98424579bd3ea8b85e5c004180636"
server_key_iv: "6600ed8fcad408d2e78b025b1b6b4af0"
server_key_version: "encryptionkey@0"
I20260810 23:31:55.750116 12347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:31:55.753535 12373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:55.754825 12347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:31:55.754990 12347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "45affa7eaf354e259778257c2d75662a"
format_stamp: "Formatted at 2026-08-10 23:31:55 on dist-test-slave-04bb"
server_key: "cee98424579bd3ea8b85e5c004180636"
server_key_iv: "6600ed8fcad408d2e78b025b1b6b4af0"
server_key_version: "encryptionkey@0"
I20260810 23:31:55.755189 12347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:55.776862 12347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:55.781216 12347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:55.781508 12347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:55.784286 12347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:31:55.784404 12347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:55.784574 12347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:31:55.784730 12347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 10 23:31:55 dist-test-slave-04bb krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404715, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:55.808768 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:55.784826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:51585 (local address 127.9.48.254:43257)
0810 23:31:55.785294 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:55.785301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:55.786625 (+ 1324us) server_negotiation.cc:407] Connection header received
0810 23:31:55.788936 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:55.788946 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:55.789050 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:55.789174 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:55.791330 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:55.792655 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:55.795469 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:55.796218 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:55.796557 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:55.801613 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:55.801620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:55.801626 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:55.801680 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:55.804959 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:55.806098 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:55.806102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:55.806105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:55.806224 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:55.806717 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:55.806723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:55.806726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:55.807004 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:55.807177 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:55.808110 (+ 933us) server_negotiation.cc:299] Negotiation successful
0810 23:31:55.808355 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":233,"threads_started":1}
I20260810 23:31:55.810639 12390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:55.785525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:43257 (local address 127.9.48.193:51585)
0810 23:31:55.786429 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:55.786483 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:55.788646 (+ 2163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:55.789430 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:55.789438 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:55.790193 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:55.791084 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:55.791113 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:55.792869 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:55.792878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:55.795240 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:55.795280 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:55.796739 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:55.796756 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:55.797081 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:55.797837 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:55.797881 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:55.801290 (+ 3409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:55.805151 (+ 3861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:55.805160 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:55.805174 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:55.805920 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:55.806341 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:55.806346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:55.806350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:55.806575 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:55.807133 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:55.807142 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:55.807728 (+ 586us) client_negotiation.cc:769] Sending connection context
0810 23:31:55.808039 (+ 311us) client_negotiation.cc:240] Negotiation successful
0810 23:31:55.808634 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":387,"threads_started":1}
I20260810 23:31:55.829375 12347 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:36737
I20260810 23:31:55.829419 12564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:36737 every 8 connection(s)
I20260810 23:31:55.830902 12347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:31:55.832875 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:55.817805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:43257 (local address 127.9.48.193:35411)
0810 23:31:55.818423 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:55.818438 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:55.818602 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:55.819128 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:55.819135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:55.819725 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:55.820030 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:55.820043 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:55.821351 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:55.821356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:55.823097 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:55.823119 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:55.824131 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:55.824140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:55.824316 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:55.825299 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:55.825318 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:55.826545 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:55.830073 (+ 3528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:55.830078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:55.830083 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:55.830484 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:55.830875 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:55.830880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:55.830884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:55.831052 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:55.831566 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:55.831571 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:55.831821 (+ 250us) client_negotiation.cc:769] Sending connection context
0810 23:31:55.832028 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 23:31:55.832384 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":379,"threads_started":1}
I20260810 23:31:55.833305 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:55.817776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:35411 (local address 127.9.48.254:43257)
0810 23:31:55.817934 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:55.817938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:55.818531 (+ 593us) server_negotiation.cc:407] Connection header received
0810 23:31:55.818724 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:55.818729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:55.818804 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:55.818902 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:55.820187 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:55.821232 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:55.823310 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:55.823922 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:55.824052 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:55.826810 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:55.826817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:55.826821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:55.826859 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:55.829934 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:55.830617 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:55.830622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:55.830626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:55.830763 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:55.831152 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:55.831157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:55.831160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:55.831443 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:55.831599 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:55.832709 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0810 23:31:55.832887 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:31:55.833420 12565 heartbeater.cc:347] Connected to a master server at 127.9.48.254:43257
I20260810 23:31:55.833665 12565 heartbeater.cc:464] Registering TS with master...
I20260810 23:31:55.834539 12565 heartbeater.cc:511] Master 127.9.48.254:43257 requested a full tablet report, sending...
I20260810 23:31:55.837044 12225 ts_manager.cc:194] Registered new tserver with Master: 45affa7eaf354e259778257c2d75662a (127.9.48.193:36737)
I20260810 23:31:55.838481 12225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:51585
I20260810 23:31:55.839962 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 12347
I20260810 23:31:55.840181 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:31:55.840911 9411 external_mini_cluster.cc:1428] Setting key e4c3ae0e7db1f9c0a1afcfea2e322c1c
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:31:55.890249 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:43257
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:34179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:56.228051 12572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:56.228432 12572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:56.228700 12572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:56.240259 12572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:56.240397 12572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:56.240559 12572 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:31:56.240646 12572 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:31:56.240752 12572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:31:56.250756 12572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:34179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.48.254:43257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:56.251961 12572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:56.253304 12572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:56.266749 12582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:56.266840 12579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:56.266887 12584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:56.268036 12572 server_base.cc:1034] running on GCE node
I20260810 23:31:56.268819 12572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:56.269819 12572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:56.271019 12572 hybrid_clock.cc:648] HybridClock initialized: now 1786404716270965 us; error 38 us; skew 500 ppm
I20260810 23:31:56.271387 12572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:56 dist-test-slave-04bb krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:56.274950 12572 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:31:56.276755 12572 webserver.cc:460] Webserver started at http://127.9.48.194:46005/ using document root <none> and password file <none>
I20260810 23:31:56.277468 12572 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:56.277581 12572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:56.277871 12572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:56.281098 12572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "eb6e4a0ea94848b5a2df256fe37536e9"
format_stamp: "Formatted at 2026-08-10 23:31:56 on dist-test-slave-04bb"
server_key: "42fe9e5b7ee7ba842c08018666d33a20"
server_key_iv: "973027838e6e5791536013be58ffdc13"
server_key_version: "encryptionkey@0"
I20260810 23:31:56.281947 12572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "eb6e4a0ea94848b5a2df256fe37536e9"
format_stamp: "Formatted at 2026-08-10 23:31:56 on dist-test-slave-04bb"
server_key: "42fe9e5b7ee7ba842c08018666d33a20"
server_key_iv: "973027838e6e5791536013be58ffdc13"
server_key_version: "encryptionkey@0"
I20260810 23:31:56.287711 12572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 23:31:56.291677 12594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:56.293303 12572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:31:56.293512 12572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "eb6e4a0ea94848b5a2df256fe37536e9"
format_stamp: "Formatted at 2026-08-10 23:31:56 on dist-test-slave-04bb"
server_key: "42fe9e5b7ee7ba842c08018666d33a20"
server_key_iv: "973027838e6e5791536013be58ffdc13"
server_key_version: "encryptionkey@0"
I20260810 23:31:56.293778 12572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:56.318228 12572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:56.322978 12572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:56.323261 12572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:56.326362 12572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:31:56.326522 12572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:56.326653 12572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:31:56.326758 12572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:31:56 dist-test-slave-04bb krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:56.352476 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:56.326556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:53805 (local address 127.9.48.254:43257)
0810 23:31:56.326760 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:56.326767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:56.328498 (+ 1731us) server_negotiation.cc:407] Connection header received
0810 23:31:56.331823 (+ 3325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:56.331832 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:56.331936 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:56.332064 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:56.335088 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.336121 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.338651 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.339451 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.339617 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:56.344449 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:56.344460 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:56.344469 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:56.344554 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:56.348062 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:56.349351 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:56.349357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:56.349361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:56.349510 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:56.350227 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:56.350234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:56.350237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:56.350578 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:56.350772 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:56.351876 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0810 23:31:56.352107 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:31:56.354031 12606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:56.327177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:43257 (local address 127.9.48.194:53805)
0810 23:31:56.328281 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:56.328350 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:56.331441 (+ 3091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:56.332393 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:56.332408 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:56.333556 (+ 1148us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:56.334841 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:56.334876 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.336291 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.336300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:56.338461 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:56.338485 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.339696 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.339701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:56.339877 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:56.340736 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:56.340775 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:56.344130 (+ 3355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:56.348231 (+ 4101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:56.348245 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:56.348277 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:56.349218 (+ 941us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:56.349646 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:56.349654 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:56.349659 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:56.350050 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:56.350875 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:56.350885 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:56.351352 (+ 467us) client_negotiation.cc:769] Sending connection context
0810 23:31:56.351756 (+ 404us) client_negotiation.cc:240] Negotiation successful
0810 23:31:56.352352 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":429,"threads_started":1}
I20260810 23:31:56.381521 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:56.365778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:35651 (local address 127.9.48.254:43257)
0810 23:31:56.366081 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:56.366091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:56.366488 (+ 397us) server_negotiation.cc:407] Connection header received
0810 23:31:56.366662 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:56.366698 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:56.366816 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:56.366994 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:56.367966 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.369566 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.371250 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.372223 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.372403 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:56.374414 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:56.374422 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:56.374427 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:56.374474 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:56.378227 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:56.379241 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:56.379247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:56.379254 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:56.379470 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:56.380019 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:56.380023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:56.380025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:56.380338 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:56.380449 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:56.381115 (+ 666us) server_negotiation.cc:299] Negotiation successful
0810 23:31:56.381284 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260810 23:31:56.382139 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:56.365562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:43257 (local address 127.9.48.194:35651)
0810 23:31:56.366366 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:56.366392 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:56.366522 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:56.367007 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:56.367011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:56.367502 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:56.367799 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:56.367810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.369738 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.369747 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:56.371060 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:56.371081 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.372421 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.372426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:56.372620 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:56.373451 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:56.373468 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:56.374170 (+ 702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:56.378411 (+ 4241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:56.378417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:56.378421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:56.379103 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:56.379622 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:56.379631 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:56.379651 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:56.379875 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:56.380463 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:56.380471 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:56.380831 (+ 360us) client_negotiation.cc:769] Sending connection context
0810 23:31:56.381025 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 23:31:56.381423 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":472,"threads_started":1}
I20260810 23:31:56.383683 12572 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:39999
I20260810 23:31:56.383715 12782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:39999 every 8 connection(s)
I20260810 23:31:56.385429 12572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:31:56.386670 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 12572
I20260810 23:31:56.386874 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:31:56.387530 9411 external_mini_cluster.cc:1428] Setting key 68d4b47154cd90ae06222bac4cf9100a
I20260810 23:31:56.387744 12783 heartbeater.cc:347] Connected to a master server at 127.9.48.254:43257
I20260810 23:31:56.388036 12783 heartbeater.cc:464] Registering TS with master...
I20260810 23:31:56.388743 12783 heartbeater.cc:511] Master 127.9.48.254:43257 requested a full tablet report, sending...
I20260810 23:31:56.390082 12225 ts_manager.cc:194] Registered new tserver with Master: eb6e4a0ea94848b5a2df256fe37536e9 (127.9.48.194:39999)
I20260810 23:31:56.391425 12225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:53805
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:31:56.436174 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:43257
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:34179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:31:56.769757 12788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:31:56.770036 12788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:31:56.770143 12788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:31:56.777318 12788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:31:56.777426 12788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:31:56.777511 12788 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:31:56.777565 12788 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:31:56.777634 12788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:31:56.787263 12788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:34179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.48.254:43257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:31:56.788518 12788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:31:56.790025 12788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:31:56.803848 12804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:56.803931 12799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:31:56.803931 12798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:31:56.804776 12788 server_base.cc:1034] running on GCE node
I20260810 23:31:56.805433 12788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:31:56.806382 12788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:31:56.807582 12788 hybrid_clock.cc:648] HybridClock initialized: now 1786404716807540 us; error 39 us; skew 500 ppm
I20260810 23:31:56.807920 12788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:31:56 dist-test-slave-04bb krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:31:56.811318 12788 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:31:56.812983 12788 webserver.cc:460] Webserver started at http://127.9.48.195:46481/ using document root <none> and password file <none>
I20260810 23:31:56.813699 12788 fs_manager.cc:359] Metadata directory not provided
I20260810 23:31:56.813831 12788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:31:56.814183 12788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:31:56.817219 12788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "acd0cc56f42a42f0ad2a448bacc43062"
format_stamp: "Formatted at 2026-08-10 23:31:56 on dist-test-slave-04bb"
server_key: "297947b41ab72617462f0ba8f1acba13"
server_key_iv: "85135cd678dcaa65c15666ef6e4f8415"
server_key_version: "encryptionkey@0"
I20260810 23:31:56.818001 12788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "acd0cc56f42a42f0ad2a448bacc43062"
format_stamp: "Formatted at 2026-08-10 23:31:56 on dist-test-slave-04bb"
server_key: "297947b41ab72617462f0ba8f1acba13"
server_key_iv: "85135cd678dcaa65c15666ef6e4f8415"
server_key_version: "encryptionkey@0"
I20260810 23:31:56.823791 12788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 23:31:56.827860 12819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:56.829649 12788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:31:56.829826 12788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "acd0cc56f42a42f0ad2a448bacc43062"
format_stamp: "Formatted at 2026-08-10 23:31:56 on dist-test-slave-04bb"
server_key: "297947b41ab72617462f0ba8f1acba13"
server_key_iv: "85135cd678dcaa65c15666ef6e4f8415"
server_key_version: "encryptionkey@0"
I20260810 23:31:56.830076 12788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:31:56.841187 12565 heartbeater.cc:503] Master 127.9.48.254:43257 was elected leader, sending a full tablet report...
I20260810 23:31:56.857117 12788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:31:56.861625 12788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:31:56.861935 12788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:31:56.864804 12788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:31:56.864933 12788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:31:56.865077 12788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:31:56.865180 12788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:31:56 dist-test-slave-04bb krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:56.888163 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:56.865246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:56745 (local address 127.9.48.254:43257)
0810 23:31:56.865443 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:56.865452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:56.866946 (+ 1494us) server_negotiation.cc:407] Connection header received
0810 23:31:56.869097 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:56.869103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:56.869203 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:56.869314 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:56.871604 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.872726 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.874332 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.874761 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.874901 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:56.879753 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:56.879761 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:56.879768 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:56.879858 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:56.883359 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:56.884654 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:56.884658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:56.884661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:56.884834 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:56.885484 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:56.885491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:56.885495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:56.885945 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:56.886124 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:56.887458 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0810 23:31:56.887732 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:31:56.890172 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:56.865894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:43257 (local address 127.9.48.195:56745)
0810 23:31:56.866785 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:56.866840 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:56.868803 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:56.869570 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:56.869578 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:56.870349 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:56.871317 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:56.871353 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.872898 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.872903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:56.874181 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:56.874195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.875028 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.875038 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:56.875411 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:56.876257 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:56.876294 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:56.879439 (+ 3145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:56.883556 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:56.883570 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:56.883591 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:56.884417 (+ 826us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:56.884982 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:56.884990 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:56.884996 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:56.885347 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:56.886248 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:56.886258 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:56.886913 (+ 655us) client_negotiation.cc:769] Sending connection context
0810 23:31:56.887315 (+ 402us) client_negotiation.cc:240] Negotiation successful
0810 23:31:56.888155 (+ 840us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":380,"threads_started":1}
I20260810 23:31:56.915751 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:56.897538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:50753 (local address 127.9.48.254:43257)
0810 23:31:56.897790 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:56.897800 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:56.898322 (+ 522us) server_negotiation.cc:407] Connection header received
0810 23:31:56.898518 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:56.898522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:56.898588 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:56.898724 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:56.900206 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.901564 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.904213 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.904895 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.905066 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:56.907487 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:56.907496 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:56.907504 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:56.907556 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:56.911609 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:56.912726 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:56.912732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:56.912739 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:56.912969 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:56.913588 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:56.913592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:56.913595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:56.913993 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:56.914165 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:56.915045 (+ 880us) server_negotiation.cc:299] Negotiation successful
0810 23:31:56.915310 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:31:56.915803 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:56.897208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:43257 (local address 127.9.48.195:50753)
0810 23:31:56.898199 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:56.898224 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:56.898391 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:56.898754 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:56.898763 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:56.899375 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:56.900021 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:56.900040 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.901728 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.901738 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:56.903956 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:56.903976 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.905406 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.905415 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:56.905652 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:56.906395 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:56.906414 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:56.907255 (+ 841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:56.911791 (+ 4536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:56.911799 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:56.911806 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:56.912473 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:56.913113 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:56.913120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:56.913127 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:56.913386 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:56.914269 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:56.914273 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:56.914577 (+ 304us) client_negotiation.cc:769] Sending connection context
0810 23:31:56.914791 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 23:31:56.915238 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":579,"threads_started":1}
I20260810 23:31:56.922119 12788 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:34805
I20260810 23:31:56.922222 13021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:34805 every 8 connection(s)
I20260810 23:31:56.923388 12788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:31:56.925534 13023 heartbeater.cc:347] Connected to a master server at 127.9.48.254:43257
I20260810 23:31:56.925768 13023 heartbeater.cc:464] Registering TS with master...
I20260810 23:31:56.926254 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 12788
I20260810 23:31:56.926455 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:31:56.926451 13023 heartbeater.cc:511] Master 127.9.48.254:43257 requested a full tablet report, sending...
I20260810 23:31:56.926965 9411 external_mini_cluster.cc:1428] Setting key 03536d9e309d0c3d6c052182db869039
I20260810 23:31:56.927887 12225 ts_manager.cc:194] Registered new tserver with Master: acd0cc56f42a42f0ad2a448bacc43062 (127.9.48.195:34805)
I20260810 23:31:56.928907 12225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:56745
I20260810 23:31:56.930540 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:31:56 dist-test-slave-04bb krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:31:56 dist-test-slave-04bb krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:56.985450 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:56.970511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.9.48.254:43257)
0810 23:31:56.970734 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:56.970738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:56.971135 (+ 397us) server_negotiation.cc:407] Connection header received
0810 23:31:56.971345 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:56.971350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:56.971427 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:56.971532 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:56.972985 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.973853 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.975025 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:56.975504 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:56.975612 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:56.979437 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:56.979442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:56.979445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:56.979480 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:56.982408 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:56.983160 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:56.983165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:56.983168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:56.983307 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:56.983767 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:56.983777 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:56.983781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:56.984076 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:56.984380 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:56.984791 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 23:31:56.985072 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 23:31:56.989815 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:31:56.997056 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:31:57.039455 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.009570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.9.48.193:36737)
0810 23:31:57.031349 (+ 21779us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:57.031359 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:57.031390 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:31:57.031502 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:57.031511 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:57.031749 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:57.032989 (+ 1240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:57.033191 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.034416 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.036919 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.038037 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.038592 (+ 555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.038820 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:57.038921 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 23:31:57.039098 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21606,"thread_start_us":21455,"threads_started":1}
I20260810 23:31:57.039485 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.009371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:36737 (local address 127.0.0.1:34114)
0810 23:31:57.018592 (+ 9221us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:57.018644 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:57.018878 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:57.032170 (+ 13292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:57.032178 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:57.032265 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:57.032778 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.032795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.034562 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.034567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.036772 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.036784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.038314 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.038321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.038591 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.038659 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 23:31:57.038777 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 23:31:57.038980 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8899,"mutex_wait_us":37,"spinlock_wait_cycles":82816,"thread_start_us":22139,"threads_started":1}
I20260810 23:31:57.039999 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.009533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:39999 (local address 127.0.0.1:49942)
0810 23:31:57.029770 (+ 20237us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:57.029787 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:57.029953 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:57.031087 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:57.031097 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:57.031142 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:57.031752 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.031773 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.033840 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.033851 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.036239 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.036256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.038482 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.038489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.039271 (+ 782us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.039310 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 23:31:57.039405 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 23:31:57.039556 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20112,"mutex_wait_us":81,"thread_start_us":20532,"threads_started":1}
I20260810 23:31:57.040581 13041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.012219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:34805 (local address 127.0.0.1:57254)
0810 23:31:57.031938 (+ 19719us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:57.031957 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:57.032263 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:57.033277 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:57.033282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:57.033306 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:57.033633 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.033644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.035415 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.035420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.037560 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.037576 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.039728 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.039735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.039882 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.039904 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:31:57.040023 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 23:31:57.040150 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19546,"mutex_wait_us":1,"spinlock_wait_cycles":1920,"thread_start_us":5895,"threads_started":1}
I20260810 23:31:57.040613 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.012700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.9.48.195:34805)
0810 23:31:57.030876 (+ 18176us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:57.030888 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:57.032086 (+ 1198us) server_negotiation.cc:407] Connection header received
0810 23:31:57.032484 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:57.032493 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:57.032752 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:57.032908 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:57.033863 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.035273 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.037761 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.039187 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.039326 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.039553 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:57.040073 (+ 520us) server_negotiation.cc:299] Negotiation successful
0810 23:31:57.040253 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17978,"thread_start_us":17770,"threads_started":1}
I20260810 23:31:57.040928 13048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.010133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.9.48.194:39999)
0810 23:31:57.028617 (+ 18484us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:57.028625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:57.029913 (+ 1288us) server_negotiation.cc:407] Connection header received
0810 23:31:57.030146 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:57.030155 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:57.030537 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:57.030776 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:57.032081 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.033651 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.036415 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.037876 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.038055 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.038327 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:57.040403 (+ 2076us) server_negotiation.cc:299] Negotiation successful
0810 23:31:57.040596 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18289,"thread_start_us":18149,"threads_started":1}
I20260810 23:31:57.044121 12460 tablet_service.cc:1467] Processing CreateTablet for tablet 11eb3a8dba374ff4bea472b7a9617247 (DEFAULT_TABLE table=test-table [id=6cfe580884234eefb6455666039b9b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:57.044389 12668 tablet_service.cc:1467] Processing CreateTablet for tablet 11eb3a8dba374ff4bea472b7a9617247 (DEFAULT_TABLE table=test-table [id=6cfe580884234eefb6455666039b9b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:57.044890 12903 tablet_service.cc:1467] Processing CreateTablet for tablet 11eb3a8dba374ff4bea472b7a9617247 (DEFAULT_TABLE table=test-table [id=6cfe580884234eefb6455666039b9b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:57.045827 12460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11eb3a8dba374ff4bea472b7a9617247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:57.045871 12668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11eb3a8dba374ff4bea472b7a9617247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:57.046259 12903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11eb3a8dba374ff4bea472b7a9617247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:57.057147 13054 tablet_bootstrap.cc:492] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a: Bootstrap starting.
I20260810 23:31:57.060662 13055 tablet_bootstrap.cc:492] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9: Bootstrap starting.
I20260810 23:31:57.061795 13054 tablet_bootstrap.cc:654] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:57.062373 13056 tablet_bootstrap.cc:492] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062: Bootstrap starting.
I20260810 23:31:57.063686 13054 log.cc:824] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:57.064445 13055 tablet_bootstrap.cc:654] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:57.065771 13056 tablet_bootstrap.cc:654] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:57.065780 13055 log.cc:824] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:57.067106 13056 log.cc:824] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062: Log is configured to *not* fsync() on all Append() calls
I20260810 23:31:57.067979 13054 tablet_bootstrap.cc:492] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a: No bootstrap required, opened a new log
I20260810 23:31:57.068266 13054 ts_tablet_manager.cc:1397] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260810 23:31:57.068862 13055 tablet_bootstrap.cc:492] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9: No bootstrap required, opened a new log
I20260810 23:31:57.069120 13055 ts_tablet_manager.cc:1397] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 23:31:57.070441 13056 tablet_bootstrap.cc:492] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062: No bootstrap required, opened a new log
I20260810 23:31:57.070706 13056 ts_tablet_manager.cc:1397] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 23:31:57.075927 13055 raft_consensus.cc:348] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.076298 13055 raft_consensus.cc:374] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:57.076422 13055 raft_consensus.cc:729] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6e4a0ea94848b5a2df256fe37536e9, State: Initialized, Role: FOLLOWER
I20260810 23:31:57.077046 13055 consensus_queue.cc:260] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.080201 13054 raft_consensus.cc:348] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.080722 13054 raft_consensus.cc:374] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:57.080837 13054 raft_consensus.cc:729] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45affa7eaf354e259778257c2d75662a, State: Initialized, Role: FOLLOWER
I20260810 23:31:57.081261 12783 heartbeater.cc:503] Master 127.9.48.254:43257 was elected leader, sending a full tablet report...
I20260810 23:31:57.081549 13054 consensus_queue.cc:260] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.082232 13055 ts_tablet_manager.cc:1428] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260810 23:31:57.082605 13056 raft_consensus.cc:348] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.083107 13056 raft_consensus.cc:374] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:57.083237 13056 raft_consensus.cc:729] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd0cc56f42a42f0ad2a448bacc43062, State: Initialized, Role: FOLLOWER
I20260810 23:31:57.084046 13054 ts_tablet_manager.cc:1428] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a: Time spent starting tablet: real 0.016s user 0.009s sys 0.009s
I20260810 23:31:57.083964 13056 consensus_queue.cc:260] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.085273 13023 heartbeater.cc:503] Master 127.9.48.254:43257 was elected leader, sending a full tablet report...
I20260810 23:31:57.085964 13056 ts_tablet_manager.cc:1428] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
W20260810 23:31:57.138365 12785 tablet.cc:2369] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:31:57.175168 13025 tablet.cc:2369] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:31:57.185564 13065 raft_consensus.cc:482] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:31:57.185885 13065 raft_consensus.cc:504] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.187634 13065 leader_election.cc:288] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45affa7eaf354e259778257c2d75662a (127.9.48.193:36737), eb6e4a0ea94848b5a2df256fe37536e9 (127.9.48.194:39999)
I20260810 23:31:57.194234 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.188186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:42767 (local address 127.9.48.193:36737)
0810 23:31:57.188379 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:57.188387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:57.188489 (+ 102us) server_negotiation.cc:407] Connection header received
0810 23:31:57.188761 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:57.188766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:57.188849 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:57.188948 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:57.189411 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.190384 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.192340 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.193390 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.193518 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.193574 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:57.193955 (+ 381us) server_negotiation.cc:299] Negotiation successful
0810 23:31:57.194050 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:31:57.194590 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.188100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:36737 (local address 127.9.48.195:42767)
0810 23:31:57.188375 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:57.188397 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:57.188586 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:57.188979 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:57.188984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:57.189016 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:57.189261 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.189275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.190528 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.190533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.192150 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.192163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.193549 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.193557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.193771 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.193816 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 23:31:57.193937 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 23:31:57.194102 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260810 23:31:57.195236 13066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.188101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:39999 (local address 127.9.48.195:35721)
0810 23:31:57.189079 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:57.189094 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:57.189207 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:57.189540 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:57.189544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:57.189569 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:57.189909 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.189922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.191206 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.191210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.192804 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.192820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.194531 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.194537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.194677 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.194700 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:31:57.194819 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 23:31:57.194953 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":254,"thread_start_us":381,"threads_started":1}
I20260810 23:31:57.195334 13048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.188162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:35721 (local address 127.9.48.194:39999)
0810 23:31:57.188368 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:57.188374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:57.189226 (+ 852us) server_negotiation.cc:407] Connection header received
0810 23:31:57.189318 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:57.189326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:57.189423 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:57.189495 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:57.190078 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.191067 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.192976 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.194210 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.194394 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.194478 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:57.194851 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 23:31:57.195028 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:31:57.195837 12488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11eb3a8dba374ff4bea472b7a9617247" candidate_uuid: "acd0cc56f42a42f0ad2a448bacc43062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45affa7eaf354e259778257c2d75662a" is_pre_election: true
I20260810 23:31:57.196296 12703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11eb3a8dba374ff4bea472b7a9617247" candidate_uuid: "acd0cc56f42a42f0ad2a448bacc43062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" is_pre_election: true
I20260810 23:31:57.196494 12488 raft_consensus.cc:2454] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd0cc56f42a42f0ad2a448bacc43062 in term 0.
I20260810 23:31:57.196892 12703 raft_consensus.cc:2454] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd0cc56f42a42f0ad2a448bacc43062 in term 0.
I20260810 23:31:57.197346 12825 leader_election.cc:302] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45affa7eaf354e259778257c2d75662a, acd0cc56f42a42f0ad2a448bacc43062; no voters:
I20260810 23:31:57.197831 13065 raft_consensus.cc:2781] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:31:57.198002 13065 raft_consensus.cc:482] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:31:57.198117 13065 raft_consensus.cc:3037] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:57.200800 13065 raft_consensus.cc:504] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.201399 13065 leader_election.cc:288] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [CANDIDATE]: Term 1 election: Requested vote from peers 45affa7eaf354e259778257c2d75662a (127.9.48.193:36737), eb6e4a0ea94848b5a2df256fe37536e9 (127.9.48.194:39999)
I20260810 23:31:57.201866 12488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11eb3a8dba374ff4bea472b7a9617247" candidate_uuid: "acd0cc56f42a42f0ad2a448bacc43062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45affa7eaf354e259778257c2d75662a"
I20260810 23:31:57.202028 12703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11eb3a8dba374ff4bea472b7a9617247" candidate_uuid: "acd0cc56f42a42f0ad2a448bacc43062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6e4a0ea94848b5a2df256fe37536e9"
I20260810 23:31:57.202092 12488 raft_consensus.cc:3037] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:57.202240 12703 raft_consensus.cc:3037] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:57.205482 12703 raft_consensus.cc:2454] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd0cc56f42a42f0ad2a448bacc43062 in term 1.
I20260810 23:31:57.205482 12488 raft_consensus.cc:2454] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd0cc56f42a42f0ad2a448bacc43062 in term 1.
I20260810 23:31:57.205914 12825 leader_election.cc:302] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45affa7eaf354e259778257c2d75662a, acd0cc56f42a42f0ad2a448bacc43062; no voters:
I20260810 23:31:57.207254 13065 raft_consensus.cc:2781] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:57.207713 13065 raft_consensus.cc:686] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [term 1 LEADER]: Becoming Leader. State: Replica: acd0cc56f42a42f0ad2a448bacc43062, State: Running, Role: LEADER
I20260810 23:31:57.208300 13065 consensus_queue.cc:237] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.213141 12222 catalog_manager.cc:5495] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 reported cstate change: term changed from 0 to 1, leader changed from <none> to acd0cc56f42a42f0ad2a448bacc43062 (127.9.48.195). New cstate: current_term: 1 leader_uuid: "acd0cc56f42a42f0ad2a448bacc43062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } health_report { overall_health: HEALTHY } } }
Aug 10 23:31:57 dist-test-slave-04bb krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404715, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:57.293462 13082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.279070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:43257 (local address 127.0.0.1:45770)
0810 23:31:57.279709 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:57.279725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:57.279894 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:57.280234 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:57.280240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:57.281026 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:31:57.281302 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.281312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.282295 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.282302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.283301 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.283314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.284214 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.284220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.284378 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.285117 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:31:57.285135 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:31:57.287441 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:31:57.290988 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:57.290999 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:57.291004 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:57.291403 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:57.291781 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:31:57.291784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:31:57.291787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:31:57.291925 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:31:57.292503 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:31:57.292511 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:31:57.292705 (+ 194us) client_negotiation.cc:769] Sending connection context
0810 23:31:57.292865 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 23:31:57.293051 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":355,"threads_started":1}
I20260810 23:31:57.293468 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.279070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45770 (local address 127.9.48.254:43257)
0810 23:31:57.279257 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:57.279262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:57.279812 (+ 550us) server_negotiation.cc:407] Connection header received
0810 23:31:57.280022 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:57.280025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:57.280097 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:57.280198 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:57.281449 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.282168 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.283513 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.283992 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.284099 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.287600 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:57.287605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:57.287609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:57.287639 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:57.290855 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:57.291513 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:57.291517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:57.291520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:57.291666 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:57.292042 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:57.292045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:57.292047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:57.292355 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:57.292581 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:57.292990 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 23:31:57.293171 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:31:57.299631 12222 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:45770:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:31:57.301280 12222 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:31:57.309990 12903 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8cdf9f41ec47af8a79d1bf18495863 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4c655e39c024dc4ba6c49f19d3e9c9d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:31:57.309990 12668 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8cdf9f41ec47af8a79d1bf18495863 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4c655e39c024dc4ba6c49f19d3e9c9d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:31:57.309870 12460 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8cdf9f41ec47af8a79d1bf18495863 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4c655e39c024dc4ba6c49f19d3e9c9d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:31:57.310982 12903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8cdf9f41ec47af8a79d1bf18495863. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:57.310982 12668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8cdf9f41ec47af8a79d1bf18495863. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:57.311044 12460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8cdf9f41ec47af8a79d1bf18495863. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:57.318445 13056 tablet_bootstrap.cc:492] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062: Bootstrap starting.
I20260810 23:31:57.320313 13056 tablet_bootstrap.cc:654] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:57.320502 13054 tablet_bootstrap.cc:492] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a: Bootstrap starting.
I20260810 23:31:57.321928 13055 tablet_bootstrap.cc:492] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9: Bootstrap starting.
I20260810 23:31:57.322244 13054 tablet_bootstrap.cc:654] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:57.323688 13055 tablet_bootstrap.cc:654] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:57.324955 13054 tablet_bootstrap.cc:492] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a: No bootstrap required, opened a new log
I20260810 23:31:57.325157 13054 ts_tablet_manager.cc:1397] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260810 23:31:57.325481 13056 tablet_bootstrap.cc:492] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062: No bootstrap required, opened a new log
I20260810 23:31:57.325630 13056 ts_tablet_manager.cc:1397] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260810 23:31:57.326220 13054 raft_consensus.cc:348] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.326550 13054 raft_consensus.cc:374] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:57.326400 13056 raft_consensus.cc:348] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.326654 13056 raft_consensus.cc:374] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:57.326776 13056 raft_consensus.cc:729] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd0cc56f42a42f0ad2a448bacc43062, State: Initialized, Role: FOLLOWER
I20260810 23:31:57.326723 13054 raft_consensus.cc:729] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45affa7eaf354e259778257c2d75662a, State: Initialized, Role: FOLLOWER
I20260810 23:31:57.327100 13054 consensus_queue.cc:260] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.327307 13056 consensus_queue.cc:260] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.327770 13065 tablet_replica.cc:439] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } } }
I20260810 23:31:57.328083 13065 tablet_replica.cc:442] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:31:57.327841 13064 tablet_replica.cc:439] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } } }
I20260810 23:31:57.328285 13064 tablet_replica.cc:442] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:31:57.328557 13056 ts_tablet_manager.cc:1428] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 23:31:57.329402 13054 ts_tablet_manager.cc:1428] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:31:57.330705 13055 tablet_bootstrap.cc:492] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9: No bootstrap required, opened a new log
I20260810 23:31:57.330860 13055 ts_tablet_manager.cc:1397] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260810 23:31:57.331720 13055 raft_consensus.cc:348] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.332000 13055 raft_consensus.cc:374] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:57.332101 13055 raft_consensus.cc:729] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6e4a0ea94848b5a2df256fe37536e9, State: Initialized, Role: FOLLOWER
I20260810 23:31:57.332347 13055 consensus_queue.cc:260] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.333078 13063 tablet_replica.cc:439] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } } }
I20260810 23:31:57.333523 13063 tablet_replica.cc:442] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 23:31:57.333986 12566 tablet.cc:2369] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:31:57.334172 13055 ts_tablet_manager.cc:1428] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:31:57.558936 13064 raft_consensus.cc:482] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:31:57.559166 13064 raft_consensus.cc:504] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.561105 13064 leader_election.cc:288] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb6e4a0ea94848b5a2df256fe37536e9 (127.9.48.194:39999), acd0cc56f42a42f0ad2a448bacc43062 (127.9.48.195:34805)
I20260810 23:31:57.570802 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.561723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:44905 (local address 127.9.48.195:34805)
0810 23:31:57.562260 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:57.562268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:57.562957 (+ 689us) server_negotiation.cc:407] Connection header received
0810 23:31:57.563167 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:57.563172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:57.563258 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:57.563356 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:57.564101 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.565262 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.568270 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.569584 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.569751 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.569817 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:57.570358 (+ 541us) server_negotiation.cc:299] Negotiation successful
0810 23:31:57.570505 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":302,"threads_started":1}
I20260810 23:31:57.570817 13048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.561613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:46719 (local address 127.9.48.194:39999)
0810 23:31:57.561834 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:57.561843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:57.562808 (+ 965us) server_negotiation.cc:407] Connection header received
0810 23:31:57.563040 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:57.563044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:57.563174 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:57.563276 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:31:57.564110 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.565284 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.568275 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.569534 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.569735 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.569817 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:57.570358 (+ 541us) server_negotiation.cc:299] Negotiation successful
0810 23:31:57.570498 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 23:31:57.571081 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.561404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:39999 (local address 127.9.48.193:46719)
0810 23:31:57.562694 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:57.562713 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:57.562872 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:57.563298 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:57.563301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:57.563348 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:57.563894 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.563910 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.565418 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.565424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.568098 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.568106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.569846 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.569854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.570110 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.570209 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 23:31:57.570317 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 23:31:57.570580 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"mutex_wait_us":274,"thread_start_us":1077,"threads_started":1}
I20260810 23:31:57.571105 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.561420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:34805 (local address 127.9.48.193:44905)
0810 23:31:57.562851 (+ 1431us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:57.562866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:57.563038 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:57.563370 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:57.563377 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:57.563401 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:31:57.563913 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.563922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.565419 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.565424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.568089 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.568105 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.569864 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.569869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.570082 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.570210 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 23:31:57.570313 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:31:57.570580 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"mutex_wait_us":488,"thread_start_us":640,"threads_started":1}
I20260810 23:31:57.571825 12703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8cdf9f41ec47af8a79d1bf18495863" candidate_uuid: "45affa7eaf354e259778257c2d75662a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" is_pre_election: true
I20260810 23:31:57.572103 12703 raft_consensus.cc:2454] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45affa7eaf354e259778257c2d75662a in term 0.
I20260810 23:31:57.572199 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8cdf9f41ec47af8a79d1bf18495863" candidate_uuid: "45affa7eaf354e259778257c2d75662a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd0cc56f42a42f0ad2a448bacc43062" is_pre_election: true
I20260810 23:31:57.572635 12945 raft_consensus.cc:2454] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45affa7eaf354e259778257c2d75662a in term 0.
I20260810 23:31:57.572646 12377 leader_election.cc:302] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45affa7eaf354e259778257c2d75662a, eb6e4a0ea94848b5a2df256fe37536e9; no voters:
I20260810 23:31:57.573057 13064 raft_consensus.cc:2781] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:31:57.573189 13064 raft_consensus.cc:482] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:31:57.573318 13064 raft_consensus.cc:3037] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:57.575520 13064 raft_consensus.cc:504] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.576138 13064 leader_election.cc:288] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [CANDIDATE]: Term 1 election: Requested vote from peers eb6e4a0ea94848b5a2df256fe37536e9 (127.9.48.194:39999), acd0cc56f42a42f0ad2a448bacc43062 (127.9.48.195:34805)
I20260810 23:31:57.576632 12703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8cdf9f41ec47af8a79d1bf18495863" candidate_uuid: "45affa7eaf354e259778257c2d75662a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6e4a0ea94848b5a2df256fe37536e9"
I20260810 23:31:57.576745 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8cdf9f41ec47af8a79d1bf18495863" candidate_uuid: "45affa7eaf354e259778257c2d75662a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd0cc56f42a42f0ad2a448bacc43062"
I20260810 23:31:57.576872 12703 raft_consensus.cc:3037] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:57.576951 12945 raft_consensus.cc:3037] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:57.579881 12703 raft_consensus.cc:2454] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45affa7eaf354e259778257c2d75662a in term 1.
I20260810 23:31:57.579895 12945 raft_consensus.cc:2454] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45affa7eaf354e259778257c2d75662a in term 1.
I20260810 23:31:57.580320 12374 leader_election.cc:302] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45affa7eaf354e259778257c2d75662a, acd0cc56f42a42f0ad2a448bacc43062; no voters:
I20260810 23:31:57.580677 13064 raft_consensus.cc:2781] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:57.581087 13064 raft_consensus.cc:686] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [term 1 LEADER]: Becoming Leader. State: Replica: 45affa7eaf354e259778257c2d75662a, State: Running, Role: LEADER
I20260810 23:31:57.581616 13064 consensus_queue.cc:237] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.585351 13094 tablet_replica.cc:439] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a: TxnStatusTablet state changed. Reason: New leader 45affa7eaf354e259778257c2d75662a. Latest consensus state: current_term: 1 leader_uuid: "45affa7eaf354e259778257c2d75662a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } } }
I20260810 23:31:57.585768 13094 tablet_replica.cc:442] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:31:57.586433 13097 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:31:57.586719 13097 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:31:57.587163 12224 catalog_manager.cc:5495] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a reported cstate change: term changed from 0 to 1, leader changed from <none> to 45affa7eaf354e259778257c2d75662a (127.9.48.193). New cstate: current_term: 1 leader_uuid: "45affa7eaf354e259778257c2d75662a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } health_report { overall_health: UNKNOWN } } }
I20260810 23:31:57.603132 13065 consensus_queue.cc:1035] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:31:57.610551 13065 consensus_queue.cc:1035] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:31:57.632510 13082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.626762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:36737 (local address 127.0.0.1:34128)
0810 23:31:57.627014 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:57.627035 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:57.627176 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:57.627699 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:57.627703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:57.627716 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:31:57.627920 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.627929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.629080 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.629086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.630086 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.630098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.630842 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.630847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.631031 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.631087 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:31:57.631950 (+ 863us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:31:57.631974 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:31:57.632059 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:31:57.632218 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 23:31:57.632570 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.626809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.9.48.193:36737)
0810 23:31:57.627112 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:57.627118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:57.627141 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:31:57.627395 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:57.627400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:57.627521 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:57.627675 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:57.628066 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.628950 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.630230 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.630677 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.630837 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.631292 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:57.631829 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:57.631995 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:57.632078 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 23:31:57.632168 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260810 23:31:57.639380 12703 raft_consensus.cc:1261] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Refusing update from remote peer 45affa7eaf354e259778257c2d75662a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:31:57.639732 12945 raft_consensus.cc:1261] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062 [term 1 FOLLOWER]: Refusing update from remote peer 45affa7eaf354e259778257c2d75662a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:31:57.640280 13094 consensus_queue.cc:1035] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:31:57.640619 13064 consensus_queue.cc:1035] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:31:57.648515 13065 tablet_replica.cc:439] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062: TxnStatusTablet state changed. Reason: New leader 45affa7eaf354e259778257c2d75662a. Latest consensus state: current_term: 1 leader_uuid: "45affa7eaf354e259778257c2d75662a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } } }
I20260810 23:31:57.648856 13065 tablet_replica.cc:442] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:31:57.652343 13105 mvcc.cc:205] Tried to move back new op lower bound from 7317113723443453952 to 7317113723217997824. Current Snapshot: MvccSnapshot[applied={T|T < 7317113723443453952}]
I20260810 23:31:57.653250 13094 tablet_replica.cc:439] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45affa7eaf354e259778257c2d75662a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } } }
I20260810 23:31:57.653522 13094 tablet_replica.cc:442] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:31:57.654819 13063 tablet_replica.cc:439] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9: TxnStatusTablet state changed. Reason: New leader 45affa7eaf354e259778257c2d75662a. Latest consensus state: current_term: 1 leader_uuid: "45affa7eaf354e259778257c2d75662a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } } }
I20260810 23:31:57.655164 13063 tablet_replica.cc:442] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:31:57.655959 13094 tablet_replica.cc:439] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45affa7eaf354e259778257c2d75662a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } } }
I20260810 23:31:57.656242 13094 tablet_replica.cc:442] T 4c8cdf9f41ec47af8a79d1bf18495863 P 45affa7eaf354e259778257c2d75662a: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:31:57.657742 13109 mvcc.cc:205] Tried to move back new op lower bound from 7317113723443453952 to 7317113723217997824. Current Snapshot: MvccSnapshot[applied={T|T < 7317113723443453952}]
I20260810 23:31:57.658527 13102 mvcc.cc:205] Tried to move back new op lower bound from 7317113723443453952 to 7317113723217997824. Current Snapshot: MvccSnapshot[applied={T|T < 7317113723443453952}]
I20260810 23:31:57.661345 13065 tablet_replica.cc:439] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45affa7eaf354e259778257c2d75662a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } } }
I20260810 23:31:57.661626 13065 tablet_replica.cc:442] T 4c8cdf9f41ec47af8a79d1bf18495863 P acd0cc56f42a42f0ad2a448bacc43062: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:31:57.671093 13063 tablet_replica.cc:439] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45affa7eaf354e259778257c2d75662a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } } }
I20260810 23:31:57.671363 13063 tablet_replica.cc:442] T 4c8cdf9f41ec47af8a79d1bf18495863 P eb6e4a0ea94848b5a2df256fe37536e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:31:57.675571 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.667555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.9.48.195:34805)
0810 23:31:57.667793 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:57.667802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:57.669154 (+ 1352us) server_negotiation.cc:407] Connection header received
0810 23:31:57.669450 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:57.669457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:57.669563 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:57.669659 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:57.670693 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.671832 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.673240 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.673878 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.674007 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.674349 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:57.675008 (+ 659us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:57.675151 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:57.675200 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 23:31:57.675283 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:31:57.691797 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.686067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:32803 (local address 127.9.48.193:36737)
0810 23:31:57.686228 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:57.686233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:57.686825 (+ 592us) server_negotiation.cc:407] Connection header received
0810 23:31:57.687070 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:57.687073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:57.687140 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:57.687234 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:57.687745 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.688585 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.689937 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.690393 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.690553 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.691003 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:57.691113 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:57.691212 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:57.691354 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 23:31:57.691438 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:31:57.691910 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.686044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:36737 (local address 127.9.48.195:32803)
0810 23:31:57.686716 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:57.686733 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:57.686901 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:57.687255 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:57.687259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:57.687271 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:31:57.687570 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.687579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.688717 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.688724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.689784 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.689796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.690589 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.690596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.690740 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.690794 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:31:57.691213 (+ 419us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:31:57.691226 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 23:31:57.691324 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 23:31:57.691471 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":356,"threads_started":1}
I20260810 23:31:57.737215 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.730073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:41549 (local address 127.9.48.195:34805)
0810 23:31:57.730251 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:57.730258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:57.731049 (+ 791us) server_negotiation.cc:407] Connection header received
0810 23:31:57.731283 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:57.731288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:57.731380 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:57.731475 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:31:57.732067 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.733041 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.735266 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.735737 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.735843 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.736406 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:31:57.736509 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:31:57.736633 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:57.736805 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 23:31:57.736893 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:31:57.737563 13136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:57.730173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:34805 (local address 127.9.48.193:41549)
0810 23:31:57.730940 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0810 23:31:57.730959 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:31:57.731136 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:31:57.731498 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:31:57.731502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:31:57.731514 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:31:57.731912 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.731922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.733188 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.733195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.735119 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:31:57.735135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:31:57.735923 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:31:57.735931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:31:57.736128 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:57.736203 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:31:57.736653 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:31:57.736666 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 23:31:57.736757 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:31:57.736909 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"spinlock_wait_cycles":49920,"thread_start_us":431,"threads_started":1}
I20260810 23:31:57.779083 12224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:31:57.779407 12224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:31:57.783676 12224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57: Sending DeleteTablet for 3 replicas of tablet 11eb3a8dba374ff4bea472b7a9617247
I20260810 23:31:57.784866 12460 tablet_service.cc:1514] Processing DeleteTablet for tablet 11eb3a8dba374ff4bea472b7a9617247 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:31:57 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:34114
I20260810 23:31:57.784946 12668 tablet_service.cc:1514] Processing DeleteTablet for tablet 11eb3a8dba374ff4bea472b7a9617247 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:31:57 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:49942
I20260810 23:31:57.787369 12903 tablet_service.cc:1514] Processing DeleteTablet for tablet 11eb3a8dba374ff4bea472b7a9617247 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:31:57 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:57254
I20260810 23:31:57.794104 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:31:57.794816 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:31:57.795280 13139 tablet_replica.cc:330] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a: stopping tablet replica
I20260810 23:31:57.795881 13139 raft_consensus.cc:2229] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:31:57.796336 13139 raft_consensus.cc:2258] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:31:57.798586 13139 ts_tablet_manager.cc:1905] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:31:57.807869 12460 tablet_service.cc:1467] Processing CreateTablet for tablet 9018dc96c69645318769052301cb975d (DEFAULT_TABLE table=test-table [id=6f8a80aba17c4313b105c21f85fd9664]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:57.808362 12460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9018dc96c69645318769052301cb975d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:57.811337 13139 ts_tablet_manager.cc:1918] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:31:57.811610 13139 log.cc:1196] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/11eb3a8dba374ff4bea472b7a9617247
I20260810 23:31:57.812382 13139 ts_tablet_manager.cc:1939] T 11eb3a8dba374ff4bea472b7a9617247 P 45affa7eaf354e259778257c2d75662a: Deleting consensus metadata
I20260810 23:31:57.814630 12198 catalog_manager.cc:4841] TS 45affa7eaf354e259778257c2d75662a (127.9.48.193:36737): tablet 11eb3a8dba374ff4bea472b7a9617247 (table test-table [id=6cfe580884234eefb6455666039b9b97]) successfully deleted
I20260810 23:31:57.815327 12902 tablet_service.cc:1467] Processing CreateTablet for tablet 9018dc96c69645318769052301cb975d (DEFAULT_TABLE table=test-table [id=6f8a80aba17c4313b105c21f85fd9664]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:57.815961 12902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9018dc96c69645318769052301cb975d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:57.817380 13140 tablet_replica.cc:330] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9: stopping tablet replica
I20260810 23:31:57.817920 13140 raft_consensus.cc:2229] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:31:57.818399 13140 raft_consensus.cc:2258] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:31:57.820545 13140 ts_tablet_manager.cc:1905] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:31:57.824471 13054 tablet_bootstrap.cc:492] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a: Bootstrap starting.
I20260810 23:31:57.824868 12667 tablet_service.cc:1467] Processing CreateTablet for tablet 9018dc96c69645318769052301cb975d (DEFAULT_TABLE table=test-table [id=6f8a80aba17c4313b105c21f85fd9664]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:57.825438 12667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9018dc96c69645318769052301cb975d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:57.826246 13054 tablet_bootstrap.cc:654] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:57.828652 13140 ts_tablet_manager.cc:1918] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:31:57.828881 13140 log.cc:1196] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/11eb3a8dba374ff4bea472b7a9617247
I20260810 23:31:57.829493 13140 ts_tablet_manager.cc:1939] T 11eb3a8dba374ff4bea472b7a9617247 P eb6e4a0ea94848b5a2df256fe37536e9: Deleting consensus metadata
I20260810 23:31:57.831255 12199 catalog_manager.cc:4841] TS eb6e4a0ea94848b5a2df256fe37536e9 (127.9.48.194:39999): tablet 11eb3a8dba374ff4bea472b7a9617247 (table test-table [id=6cfe580884234eefb6455666039b9b97]) successfully deleted
I20260810 23:31:57.845203 13146 tablet_bootstrap.cc:492] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9: Bootstrap starting.
I20260810 23:31:57.846652 13145 tablet_bootstrap.cc:492] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062: Bootstrap starting.
I20260810 23:31:57.847851 13054 tablet_bootstrap.cc:492] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a: No bootstrap required, opened a new log
I20260810 23:31:57.848071 13054 ts_tablet_manager.cc:1397] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.006s
I20260810 23:31:57.848846 13145 tablet_bootstrap.cc:654] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:57.849066 13054 raft_consensus.cc:348] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.849408 13054 raft_consensus.cc:374] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:57.849519 13054 raft_consensus.cc:729] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45affa7eaf354e259778257c2d75662a, State: Initialized, Role: FOLLOWER
I20260810 23:31:57.850507 13054 consensus_queue.cc:260] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.851048 13146 tablet_bootstrap.cc:654] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:57.851392 13054 ts_tablet_manager.cc:1428] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:31:57.852510 13143 tablet_replica.cc:330] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062: stopping tablet replica
I20260810 23:31:57.853032 13143 raft_consensus.cc:2229] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:31:57.853787 13143 raft_consensus.cc:2258] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:31:57.856227 13143 ts_tablet_manager.cc:1905] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:31:57.860167 13146 tablet_bootstrap.cc:492] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9: No bootstrap required, opened a new log
I20260810 23:31:57.860348 13146 ts_tablet_manager.cc:1397] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260810 23:31:57.861235 13146 raft_consensus.cc:348] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.861500 13146 raft_consensus.cc:374] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:57.861626 13146 raft_consensus.cc:729] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6e4a0ea94848b5a2df256fe37536e9, State: Initialized, Role: FOLLOWER
I20260810 23:31:57.861939 13146 consensus_queue.cc:260] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.863868 13145 tablet_bootstrap.cc:492] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062: No bootstrap required, opened a new log
I20260810 23:31:57.864058 13145 ts_tablet_manager.cc:1397] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20260810 23:31:57.865072 13145 raft_consensus.cc:348] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.865401 13145 raft_consensus.cc:374] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:57.865507 13146 ts_tablet_manager.cc:1428] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:31:57.865522 13145 raft_consensus.cc:729] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd0cc56f42a42f0ad2a448bacc43062, State: Initialized, Role: FOLLOWER
I20260810 23:31:57.865800 13145 consensus_queue.cc:260] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:57.866395 13143 ts_tablet_manager.cc:1918] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:31:57.866463 13145 ts_tablet_manager.cc:1428] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:31:57.866598 13143 log.cc:1196] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/11eb3a8dba374ff4bea472b7a9617247
I20260810 23:31:57.867365 13143 ts_tablet_manager.cc:1939] T 11eb3a8dba374ff4bea472b7a9617247 P acd0cc56f42a42f0ad2a448bacc43062: Deleting consensus metadata
I20260810 23:31:57.869429 12197 catalog_manager.cc:4841] TS acd0cc56f42a42f0ad2a448bacc43062 (127.9.48.195:34805): tablet 11eb3a8dba374ff4bea472b7a9617247 (table test-table [id=6cfe580884234eefb6455666039b9b97]) successfully deleted
I20260810 23:31:58.134734 13065 raft_consensus.cc:482] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:31:58.134981 13065 raft_consensus.cc:504] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:58.135854 13065 leader_election.cc:288] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45affa7eaf354e259778257c2d75662a (127.9.48.193:36737), eb6e4a0ea94848b5a2df256fe37536e9 (127.9.48.194:39999)
I20260810 23:31:58.136472 12488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9018dc96c69645318769052301cb975d" candidate_uuid: "acd0cc56f42a42f0ad2a448bacc43062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45affa7eaf354e259778257c2d75662a" is_pre_election: true
I20260810 23:31:58.136547 12702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9018dc96c69645318769052301cb975d" candidate_uuid: "acd0cc56f42a42f0ad2a448bacc43062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" is_pre_election: true
I20260810 23:31:58.136786 12488 raft_consensus.cc:2454] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd0cc56f42a42f0ad2a448bacc43062 in term 0.
I20260810 23:31:58.136804 12702 raft_consensus.cc:2454] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd0cc56f42a42f0ad2a448bacc43062 in term 0.
I20260810 23:31:58.137305 12825 leader_election.cc:302] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45affa7eaf354e259778257c2d75662a, acd0cc56f42a42f0ad2a448bacc43062; no voters:
I20260810 23:31:58.137662 13065 raft_consensus.cc:2781] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:31:58.137776 13065 raft_consensus.cc:482] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:31:58.137923 13065 raft_consensus.cc:3037] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:58.140172 13065 raft_consensus.cc:504] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:58.140865 13065 leader_election.cc:288] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [CANDIDATE]: Term 1 election: Requested vote from peers 45affa7eaf354e259778257c2d75662a (127.9.48.193:36737), eb6e4a0ea94848b5a2df256fe37536e9 (127.9.48.194:39999)
I20260810 23:31:58.141428 12702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9018dc96c69645318769052301cb975d" candidate_uuid: "acd0cc56f42a42f0ad2a448bacc43062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6e4a0ea94848b5a2df256fe37536e9"
I20260810 23:31:58.141640 12702 raft_consensus.cc:3037] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:58.141707 12488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9018dc96c69645318769052301cb975d" candidate_uuid: "acd0cc56f42a42f0ad2a448bacc43062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45affa7eaf354e259778257c2d75662a"
I20260810 23:31:58.141973 12488 raft_consensus.cc:3037] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:31:58.143910 12702 raft_consensus.cc:2454] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd0cc56f42a42f0ad2a448bacc43062 in term 1.
I20260810 23:31:58.144351 12826 leader_election.cc:302] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acd0cc56f42a42f0ad2a448bacc43062, eb6e4a0ea94848b5a2df256fe37536e9; no voters:
I20260810 23:31:58.144666 13065 raft_consensus.cc:2781] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:31:58.144902 13065 raft_consensus.cc:686] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [term 1 LEADER]: Becoming Leader. State: Replica: acd0cc56f42a42f0ad2a448bacc43062, State: Running, Role: LEADER
I20260810 23:31:58.145103 12488 raft_consensus.cc:2454] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd0cc56f42a42f0ad2a448bacc43062 in term 1.
I20260810 23:31:58.145169 13065 consensus_queue.cc:237] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:58.148147 12225 catalog_manager.cc:5495] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 reported cstate change: term changed from 0 to 1, leader changed from <none> to acd0cc56f42a42f0ad2a448bacc43062 (127.9.48.195). New cstate: current_term: 1 leader_uuid: "acd0cc56f42a42f0ad2a448bacc43062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } health_report { overall_health: HEALTHY } } }
I20260810 23:31:58.227916 12702 raft_consensus.cc:1261] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Refusing update from remote peer acd0cc56f42a42f0ad2a448bacc43062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:31:58.228981 13106 consensus_queue.cc:1035] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:31:58.229277 12488 raft_consensus.cc:1261] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Refusing update from remote peer acd0cc56f42a42f0ad2a448bacc43062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:31:58.229849 13106 consensus_queue.cc:1035] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:31:58.263403 13102 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 23:31:59.872201 12224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:31:59.872439 12224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:31:59.877048 12224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57: Sending DeleteTablet for 3 replicas of tablet 9018dc96c69645318769052301cb975d
I20260810 23:31:59.878014 12460 tablet_service.cc:1514] Processing DeleteTablet for tablet 9018dc96c69645318769052301cb975d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:31:59 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:34114
I20260810 23:31:59.878005 12667 tablet_service.cc:1514] Processing DeleteTablet for tablet 9018dc96c69645318769052301cb975d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:31:59 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:49942
I20260810 23:31:59.878211 12903 tablet_service.cc:1514] Processing DeleteTablet for tablet 9018dc96c69645318769052301cb975d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:31:59 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:57254
I20260810 23:31:59.890375 13203 tablet_replica.cc:330] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a: stopping tablet replica
I20260810 23:31:59.890784 13203 raft_consensus.cc:2229] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:31:59.891119 13203 raft_consensus.cc:2258] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:31:59.891777 13204 tablet_replica.cc:330] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062: stopping tablet replica
I20260810 23:31:59.892055 13204 raft_consensus.cc:2229] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:31:59.892100 13201 tablet_replica.cc:330] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9: stopping tablet replica
I20260810 23:31:59.892313 13201 raft_consensus.cc:2229] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:31:59.892647 13201 raft_consensus.cc:2258] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:31:59.892625 13204 raft_consensus.cc:2258] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:31:59.892712 13203 ts_tablet_manager.cc:1905] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:31:59.894071 13201 ts_tablet_manager.cc:1905] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:31:59.894832 13204 ts_tablet_manager.cc:1905] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:31:59.899793 13203 ts_tablet_manager.cc:1918] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:31:59.900002 13203 log.cc:1196] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/9018dc96c69645318769052301cb975d
I20260810 23:31:59.900543 13203 ts_tablet_manager.cc:1939] T 9018dc96c69645318769052301cb975d P 45affa7eaf354e259778257c2d75662a: Deleting consensus metadata
I20260810 23:31:59.900669 13201 ts_tablet_manager.cc:1918] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:31:59.900816 13201 log.cc:1196] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/9018dc96c69645318769052301cb975d
I20260810 23:31:59.901260 13201 ts_tablet_manager.cc:1939] T 9018dc96c69645318769052301cb975d P eb6e4a0ea94848b5a2df256fe37536e9: Deleting consensus metadata
I20260810 23:31:59.901741 13204 ts_tablet_manager.cc:1918] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:31:59.901861 12198 catalog_manager.cc:4841] TS 45affa7eaf354e259778257c2d75662a (127.9.48.193:36737): tablet 9018dc96c69645318769052301cb975d (table test-table [id=6f8a80aba17c4313b105c21f85fd9664]) successfully deleted
I20260810 23:31:59.902019 13204 log.cc:1196] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/9018dc96c69645318769052301cb975d
I20260810 23:31:59.902485 13204 ts_tablet_manager.cc:1939] T 9018dc96c69645318769052301cb975d P acd0cc56f42a42f0ad2a448bacc43062: Deleting consensus metadata
I20260810 23:31:59.902659 12199 catalog_manager.cc:4841] TS eb6e4a0ea94848b5a2df256fe37536e9 (127.9.48.194:39999): tablet 9018dc96c69645318769052301cb975d (table test-table [id=6f8a80aba17c4313b105c21f85fd9664]) successfully deleted
I20260810 23:31:59.903501 12197 catalog_manager.cc:4841] TS acd0cc56f42a42f0ad2a448bacc43062 (127.9.48.195:34805): tablet 9018dc96c69645318769052301cb975d (table test-table [id=6f8a80aba17c4313b105c21f85fd9664]) successfully deleted
Aug 10 23:31:59 dist-test-slave-04bb krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:31:59 dist-test-slave-04bb krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:31:59.950373 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:31:59.929390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.9.48.254:43257)
0810 23:31:59.930286 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0810 23:31:59.930291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:31:59.930312 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:31:59.930410 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:31:59.930415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:31:59.930493 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:31:59.930616 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:31:59.932296 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:59.933171 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:59.935005 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:31:59.935473 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:31:59.935601 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:31:59.945019 (+ 9418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:31:59.945024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:31:59.945028 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:31:59.945063 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:31:59.947833 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:59.948511 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:59.948516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:59.948518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:59.948712 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:31:59.949072 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:31:59.949076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:31:59.949078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:31:59.949373 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:31:59.949556 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 23:31:59.949898 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 23:31:59.950076 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":631,"threads_started":1}
I20260810 23:31:59.954092 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:31:59.954726 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:31:59.961457 12460 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8d9b43fbb64d6e8d35dc41acaa2641 (DEFAULT_TABLE table=test-table [id=f54975005265406b9484c9e78c019e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:59.961575 12667 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8d9b43fbb64d6e8d35dc41acaa2641 (DEFAULT_TABLE table=test-table [id=f54975005265406b9484c9e78c019e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:59.961974 12460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8d9b43fbb64d6e8d35dc41acaa2641. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:59.962050 12667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8d9b43fbb64d6e8d35dc41acaa2641. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:59.961954 12903 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8d9b43fbb64d6e8d35dc41acaa2641 (DEFAULT_TABLE table=test-table [id=f54975005265406b9484c9e78c019e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:31:59.962415 12903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f8d9b43fbb64d6e8d35dc41acaa2641. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:31:59.968716 13221 tablet_bootstrap.cc:492] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a: Bootstrap starting.
I20260810 23:31:59.970053 13221 tablet_bootstrap.cc:654] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:59.973166 13221 tablet_bootstrap.cc:492] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a: No bootstrap required, opened a new log
I20260810 23:31:59.973333 13221 ts_tablet_manager.cc:1397] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:31:59.973958 13221 raft_consensus.cc:348] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:59.974160 13221 raft_consensus.cc:374] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:59.974270 13221 raft_consensus.cc:729] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45affa7eaf354e259778257c2d75662a, State: Initialized, Role: FOLLOWER
I20260810 23:31:59.974524 13221 consensus_queue.cc:260] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:59.975064 13221 ts_tablet_manager.cc:1428] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 23:31:59.975385 13223 tablet_bootstrap.cc:492] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9: Bootstrap starting.
I20260810 23:31:59.977094 13223 tablet_bootstrap.cc:654] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:59.977918 13224 tablet_bootstrap.cc:492] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062: Bootstrap starting.
I20260810 23:31:59.979427 13223 tablet_bootstrap.cc:492] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9: No bootstrap required, opened a new log
I20260810 23:31:59.979573 13223 ts_tablet_manager.cc:1397] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260810 23:31:59.979703 13224 tablet_bootstrap.cc:654] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062: Neither blocks nor log segments found. Creating new log.
I20260810 23:31:59.980376 13223 raft_consensus.cc:348] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:59.980656 13223 raft_consensus.cc:374] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:59.980779 13223 raft_consensus.cc:729] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6e4a0ea94848b5a2df256fe37536e9, State: Initialized, Role: FOLLOWER
I20260810 23:31:59.981065 13223 consensus_queue.cc:260] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:59.981917 13223 ts_tablet_manager.cc:1428] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 23:31:59.982398 13224 tablet_bootstrap.cc:492] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062: No bootstrap required, opened a new log
I20260810 23:31:59.982561 13224 ts_tablet_manager.cc:1397] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:31:59.983403 13224 raft_consensus.cc:348] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:59.983659 13224 raft_consensus.cc:374] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:31:59.983770 13224 raft_consensus.cc:729] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd0cc56f42a42f0ad2a448bacc43062, State: Initialized, Role: FOLLOWER
I20260810 23:31:59.984046 13224 consensus_queue.cc:260] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:31:59.984668 13224 ts_tablet_manager.cc:1428] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 23:32:00.242990 13238 raft_consensus.cc:482] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:00.243289 13238 raft_consensus.cc:504] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:32:00.245052 13238 leader_election.cc:288] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45affa7eaf354e259778257c2d75662a (127.9.48.193:36737), acd0cc56f42a42f0ad2a448bacc43062 (127.9.48.195:34805)
I20260810 23:32:00.251827 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:00.245435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:58043 (local address 127.9.48.193:36737)
0810 23:32:00.246182 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:00.246190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:00.246731 (+ 541us) server_negotiation.cc:407] Connection header received
0810 23:32:00.246816 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:00.246820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:00.246879 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:00.246985 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:00.247544 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.248327 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.250092 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.250834 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.251031 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:00.251109 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:00.251364 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 23:32:00.251491 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":372,"threads_started":1}
I20260810 23:32:00.251904 13243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:00.245426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:36737 (local address 127.9.48.194:58043)
0810 23:32:00.246427 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:00.246454 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:00.246636 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:00.247007 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:00.247011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:00.247039 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:00.247357 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:00.247366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.248458 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.248465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:00.249921 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:00.249933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.251028 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.251034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:00.251189 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:00.251212 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:32:00.251314 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:32:00.251440 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":1011,"threads_started":1}
I20260810 23:32:00.252583 12488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8d9b43fbb64d6e8d35dc41acaa2641" candidate_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45affa7eaf354e259778257c2d75662a" is_pre_election: true
I20260810 23:32:00.252864 12488 raft_consensus.cc:2454] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6e4a0ea94848b5a2df256fe37536e9 in term 0.
I20260810 23:32:00.253000 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:00.245674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:34805 (local address 127.9.48.194:45355)
0810 23:32:00.246781 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:00.246803 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:00.246964 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:00.247353 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:00.247358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:00.247389 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:00.247777 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:00.247787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.249049 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.249058 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:00.250569 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:00.250582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.252070 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.252080 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:00.252256 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:00.252281 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:32:00.252397 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 23:32:00.252534 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":1,"thread_start_us":397,"threads_started":1}
I20260810 23:32:00.252978 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:00.245866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:45355 (local address 127.9.48.195:34805)
0810 23:32:00.246807 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:00.246814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:00.246902 (+ 88us) server_negotiation.cc:407] Connection header received
0810 23:32:00.247119 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:00.247123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:00.247205 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:00.247320 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:00.247983 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.248899 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.250753 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.251596 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.251829 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:00.251935 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:00.252428 (+ 493us) server_negotiation.cc:299] Negotiation successful
0810 23:32:00.252619 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":735,"threads_started":1}
I20260810 23:32:00.253450 12598 leader_election.cc:302] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45affa7eaf354e259778257c2d75662a, eb6e4a0ea94848b5a2df256fe37536e9; no voters:
I20260810 23:32:00.253645 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8d9b43fbb64d6e8d35dc41acaa2641" candidate_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd0cc56f42a42f0ad2a448bacc43062" is_pre_election: true
I20260810 23:32:00.253831 13238 raft_consensus.cc:2781] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:00.253964 13238 raft_consensus.cc:482] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:00.253937 12945 raft_consensus.cc:2454] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6e4a0ea94848b5a2df256fe37536e9 in term 0.
I20260810 23:32:00.254050 13238 raft_consensus.cc:3037] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:00.256714 13238 raft_consensus.cc:504] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:32:00.257391 13238 leader_election.cc:288] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [CANDIDATE]: Term 1 election: Requested vote from peers 45affa7eaf354e259778257c2d75662a (127.9.48.193:36737), acd0cc56f42a42f0ad2a448bacc43062 (127.9.48.195:34805)
I20260810 23:32:00.257700 12488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8d9b43fbb64d6e8d35dc41acaa2641" candidate_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45affa7eaf354e259778257c2d75662a"
I20260810 23:32:00.257936 12488 raft_consensus.cc:3037] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:00.257943 12945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f8d9b43fbb64d6e8d35dc41acaa2641" candidate_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd0cc56f42a42f0ad2a448bacc43062"
I20260810 23:32:00.258176 12945 raft_consensus.cc:3037] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:00.260329 12488 raft_consensus.cc:2454] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6e4a0ea94848b5a2df256fe37536e9 in term 1.
I20260810 23:32:00.260346 12945 raft_consensus.cc:2454] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6e4a0ea94848b5a2df256fe37536e9 in term 1.
I20260810 23:32:00.260751 12597 leader_election.cc:302] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acd0cc56f42a42f0ad2a448bacc43062, eb6e4a0ea94848b5a2df256fe37536e9; no voters:
I20260810 23:32:00.261054 13238 raft_consensus.cc:2781] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:00.261551 13238 raft_consensus.cc:686] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 LEADER]: Becoming Leader. State: Replica: eb6e4a0ea94848b5a2df256fe37536e9, State: Running, Role: LEADER
I20260810 23:32:00.261963 13238 consensus_queue.cc:237] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:32:00.265228 12225 catalog_manager.cc:5495] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6e4a0ea94848b5a2df256fe37536e9 (127.9.48.194). New cstate: current_term: 1 leader_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } health_report { overall_health: UNKNOWN } } }
I20260810 23:32:00.346025 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:00.340129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.9.48.194:39999)
0810 23:32:00.340794 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:00.340800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:00.340827 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:32:00.340934 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:00.340937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:00.341017 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:00.341163 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:32:00.341534 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.342390 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.343755 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.344137 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.344236 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:00.344736 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:32:00.345353 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:32:00.345480 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:00.345577 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 23:32:00.345641 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":324,"threads_started":1}
I20260810 23:32:00.361395 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:00.355495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:54385 (local address 127.9.48.193:36737)
0810 23:32:00.355650 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:00.355655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:00.356306 (+ 651us) server_negotiation.cc:407] Connection header received
0810 23:32:00.356569 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:00.356586 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:00.356668 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:00.356754 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:32:00.357207 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.358063 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.359512 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.360024 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.360129 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:00.360688 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:32:00.360786 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:32:00.360859 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:00.360977 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 23:32:00.361043 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:32:00.361445 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:00.355502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:36737 (local address 127.9.48.194:54385)
0810 23:32:00.356194 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:00.356222 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:00.356378 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:00.356773 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:00.356777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:00.356787 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:32:00.357034 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:00.357040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.358210 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.358214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:00.359375 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:00.359387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.360246 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.360251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:00.360396 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:00.360447 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:32:00.360874 (+ 427us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:32:00.360883 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:32:00.360944 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 23:32:00.361044 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":342,"threads_started":1}
I20260810 23:32:00.369294 12945 raft_consensus.cc:1261] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062 [term 1 FOLLOWER]: Refusing update from remote peer eb6e4a0ea94848b5a2df256fe37536e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:00.369289 12488 raft_consensus.cc:1261] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Refusing update from remote peer eb6e4a0ea94848b5a2df256fe37536e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:00.369863 13247 consensus_queue.cc:1035] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:00.370265 13238 consensus_queue.cc:1035] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:00.414378 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:00.395631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:33131 (local address 127.9.48.194:39999)
0810 23:32:00.395872 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:00.395878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:00.409118 (+ 13240us) server_negotiation.cc:407] Connection header received
0810 23:32:00.409331 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:00.409337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:00.409416 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:00.409498 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:32:00.409901 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.410983 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.412301 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.413059 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.413194 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:00.413609 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:32:00.413721 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:32:00.413819 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:00.413994 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 23:32:00.414069 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260810 23:32:00.414485 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:00.395601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:39999 (local address 127.9.48.193:33131)
0810 23:32:00.408991 (+ 13390us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:00.409008 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:00.409181 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:00.409536 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:00.409540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:00.409551 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:32:00.409772 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:00.409780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.411131 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.411135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:00.412113 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:00.412125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.413240 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.413249 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:00.413401 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:00.413439 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:32:00.413858 (+ 419us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:32:00.413870 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 23:32:00.413964 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 23:32:00.414102 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13224,"thread_start_us":13109,"threads_started":1}
I20260810 23:32:00.448134 12224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:00.448346 12224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:00.451608 12224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57: Sending DeleteTablet for 3 replicas of tablet 7f8d9b43fbb64d6e8d35dc41acaa2641
I20260810 23:32:00.452422 12460 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8d9b43fbb64d6e8d35dc41acaa2641 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:00 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:34114
I20260810 23:32:00.452503 12667 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8d9b43fbb64d6e8d35dc41acaa2641 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:00 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:49942
I20260810 23:32:00.453263 12903 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8d9b43fbb64d6e8d35dc41acaa2641 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:00 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:57254
I20260810 23:32:00.453866 12224 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:00.454670 12224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:00.463599 12459 tablet_service.cc:1467] Processing CreateTablet for tablet 6f67a27634aa4e4db57d16d8c2e7edff (DEFAULT_TABLE table=test-table [id=0af47bba06f4425ca1bcd0d1ddde770d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:00.463711 12668 tablet_service.cc:1467] Processing CreateTablet for tablet 6f67a27634aa4e4db57d16d8c2e7edff (DEFAULT_TABLE table=test-table [id=0af47bba06f4425ca1bcd0d1ddde770d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:00.464181 12459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f67a27634aa4e4db57d16d8c2e7edff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:00.464191 12668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f67a27634aa4e4db57d16d8c2e7edff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:00.464982 13272 tablet_replica.cc:330] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9: stopping tablet replica
I20260810 23:32:00.465170 13272 raft_consensus.cc:2229] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:00.465776 13272 raft_consensus.cc:2258] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:00.467347 13272 ts_tablet_manager.cc:1905] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:00.470263 12902 tablet_service.cc:1467] Processing CreateTablet for tablet 6f67a27634aa4e4db57d16d8c2e7edff (DEFAULT_TABLE table=test-table [id=0af47bba06f4425ca1bcd0d1ddde770d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:00.470726 12902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f67a27634aa4e4db57d16d8c2e7edff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:00.471870 13221 tablet_bootstrap.cc:492] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a: Bootstrap starting.
I20260810 23:32:00.473795 13221 tablet_bootstrap.cc:654] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:00.475108 13275 tablet_replica.cc:330] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a: stopping tablet replica
I20260810 23:32:00.478156 13272 ts_tablet_manager.cc:1918] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:32:00.478555 13275 raft_consensus.cc:2229] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:00.478956 13275 raft_consensus.cc:2258] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:00.479344 13272 log.cc:1196] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/7f8d9b43fbb64d6e8d35dc41acaa2641
I20260810 23:32:00.480057 13272 ts_tablet_manager.cc:1939] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P eb6e4a0ea94848b5a2df256fe37536e9: Deleting consensus metadata
I20260810 23:32:00.481567 12199 catalog_manager.cc:4841] TS eb6e4a0ea94848b5a2df256fe37536e9 (127.9.48.194:39999): tablet 7f8d9b43fbb64d6e8d35dc41acaa2641 (table test-table [id=f54975005265406b9484c9e78c019e43]) successfully deleted
I20260810 23:32:00.483417 13221 tablet_bootstrap.cc:492] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a: No bootstrap required, opened a new log
I20260810 23:32:00.483587 13221 ts_tablet_manager.cc:1397] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20260810 23:32:00.484095 13275 ts_tablet_manager.cc:1905] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:00.484463 13221 raft_consensus.cc:348] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:32:00.484777 13221 raft_consensus.cc:374] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:00.484869 13221 raft_consensus.cc:729] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45affa7eaf354e259778257c2d75662a, State: Initialized, Role: FOLLOWER
I20260810 23:32:00.485057 13224 tablet_bootstrap.cc:492] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062: Bootstrap starting.
I20260810 23:32:00.485240 13221 consensus_queue.cc:260] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:32:00.485749 13281 tablet_bootstrap.cc:492] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9: Bootstrap starting.
I20260810 23:32:00.486807 13224 tablet_bootstrap.cc:654] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:00.487437 13281 tablet_bootstrap.cc:654] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:00.488258 13221 ts_tablet_manager.cc:1428] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260810 23:32:00.492973 13276 tablet_replica.cc:330] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062: stopping tablet replica
I20260810 23:32:00.493008 13275 ts_tablet_manager.cc:1918] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:32:00.493229 13275 log.cc:1196] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/7f8d9b43fbb64d6e8d35dc41acaa2641
I20260810 23:32:00.493228 13276 raft_consensus.cc:2229] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:00.493438 13276 raft_consensus.cc:2258] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:00.493896 13275 ts_tablet_manager.cc:1939] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P 45affa7eaf354e259778257c2d75662a: Deleting consensus metadata
I20260810 23:32:00.494447 13276 ts_tablet_manager.cc:1905] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:00.494999 12198 catalog_manager.cc:4841] TS 45affa7eaf354e259778257c2d75662a (127.9.48.193:36737): tablet 7f8d9b43fbb64d6e8d35dc41acaa2641 (table test-table [id=f54975005265406b9484c9e78c019e43]) successfully deleted
I20260810 23:32:00.498594 13281 tablet_bootstrap.cc:492] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9: No bootstrap required, opened a new log
I20260810 23:32:00.498766 13281 ts_tablet_manager.cc:1397] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260810 23:32:00.499601 13281 raft_consensus.cc:348] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:32:00.499871 13281 raft_consensus.cc:374] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:00.499984 13281 raft_consensus.cc:729] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6e4a0ea94848b5a2df256fe37536e9, State: Initialized, Role: FOLLOWER
I20260810 23:32:00.500253 13281 consensus_queue.cc:260] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:32:00.502799 13224 tablet_bootstrap.cc:492] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062: No bootstrap required, opened a new log
I20260810 23:32:00.502923 13276 ts_tablet_manager.cc:1918] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:32:00.503060 13276 log.cc:1196] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/7f8d9b43fbb64d6e8d35dc41acaa2641
I20260810 23:32:00.503070 13224 ts_tablet_manager.cc:1397] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.010s
I20260810 23:32:00.503608 13276 ts_tablet_manager.cc:1939] T 7f8d9b43fbb64d6e8d35dc41acaa2641 P acd0cc56f42a42f0ad2a448bacc43062: Deleting consensus metadata
I20260810 23:32:00.504107 13224 raft_consensus.cc:348] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:32:00.504266 13281 ts_tablet_manager.cc:1428] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:32:00.504441 13224 raft_consensus.cc:374] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:00.504554 13224 raft_consensus.cc:729] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd0cc56f42a42f0ad2a448bacc43062, State: Initialized, Role: FOLLOWER
I20260810 23:32:00.504819 13224 consensus_queue.cc:260] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:32:00.504856 12197 catalog_manager.cc:4841] TS acd0cc56f42a42f0ad2a448bacc43062 (127.9.48.195:34805): tablet 7f8d9b43fbb64d6e8d35dc41acaa2641 (table test-table [id=f54975005265406b9484c9e78c019e43]) successfully deleted
I20260810 23:32:00.505344 13224 ts_tablet_manager.cc:1428] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260810 23:32:00.571447 13160 raft_consensus.cc:482] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:00.571655 13160 raft_consensus.cc:504] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:32:00.572373 13160 leader_election.cc:288] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45affa7eaf354e259778257c2d75662a (127.9.48.193:36737), eb6e4a0ea94848b5a2df256fe37536e9 (127.9.48.194:39999)
I20260810 23:32:00.572983 12488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f67a27634aa4e4db57d16d8c2e7edff" candidate_uuid: "acd0cc56f42a42f0ad2a448bacc43062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45affa7eaf354e259778257c2d75662a" is_pre_election: true
I20260810 23:32:00.573163 12702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f67a27634aa4e4db57d16d8c2e7edff" candidate_uuid: "acd0cc56f42a42f0ad2a448bacc43062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" is_pre_election: true
I20260810 23:32:00.573280 12488 raft_consensus.cc:2454] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd0cc56f42a42f0ad2a448bacc43062 in term 0.
I20260810 23:32:00.573454 12702 raft_consensus.cc:2454] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd0cc56f42a42f0ad2a448bacc43062 in term 0.
I20260810 23:32:00.573781 12825 leader_election.cc:302] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45affa7eaf354e259778257c2d75662a, acd0cc56f42a42f0ad2a448bacc43062; no voters:
I20260810 23:32:00.574100 13160 raft_consensus.cc:2781] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:00.574237 13160 raft_consensus.cc:482] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:00.574337 13160 raft_consensus.cc:3037] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:00.577544 13160 raft_consensus.cc:504] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:32:00.578251 13160 leader_election.cc:288] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [CANDIDATE]: Term 1 election: Requested vote from peers 45affa7eaf354e259778257c2d75662a (127.9.48.193:36737), eb6e4a0ea94848b5a2df256fe37536e9 (127.9.48.194:39999)
I20260810 23:32:00.578778 12488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f67a27634aa4e4db57d16d8c2e7edff" candidate_uuid: "acd0cc56f42a42f0ad2a448bacc43062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45affa7eaf354e259778257c2d75662a"
I20260810 23:32:00.578805 12702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f67a27634aa4e4db57d16d8c2e7edff" candidate_uuid: "acd0cc56f42a42f0ad2a448bacc43062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb6e4a0ea94848b5a2df256fe37536e9"
I20260810 23:32:00.579022 12488 raft_consensus.cc:3037] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:00.579046 12702 raft_consensus.cc:3037] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:00.581369 12702 raft_consensus.cc:2454] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd0cc56f42a42f0ad2a448bacc43062 in term 1.
I20260810 23:32:00.581820 12826 leader_election.cc:302] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acd0cc56f42a42f0ad2a448bacc43062, eb6e4a0ea94848b5a2df256fe37536e9; no voters:
I20260810 23:32:00.582099 12488 raft_consensus.cc:2454] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd0cc56f42a42f0ad2a448bacc43062 in term 1.
I20260810 23:32:00.582154 13160 raft_consensus.cc:2781] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:00.582854 13160 raft_consensus.cc:686] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [term 1 LEADER]: Becoming Leader. State: Replica: acd0cc56f42a42f0ad2a448bacc43062, State: Running, Role: LEADER
I20260810 23:32:00.583168 13160 consensus_queue.cc:237] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } }
I20260810 23:32:00.586305 12225 catalog_manager.cc:5495] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 reported cstate change: term changed from 0 to 1, leader changed from <none> to acd0cc56f42a42f0ad2a448bacc43062 (127.9.48.195). New cstate: current_term: 1 leader_uuid: "acd0cc56f42a42f0ad2a448bacc43062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acd0cc56f42a42f0ad2a448bacc43062" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34805 } health_report { overall_health: HEALTHY } } }
I20260810 23:32:00.614527 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:00.608600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.9.48.195:34805)
0810 23:32:00.608835 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:00.608842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:00.609152 (+ 310us) server_negotiation.cc:407] Connection header received
0810 23:32:00.609321 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:00.609325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:00.609390 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:00.609471 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:32:00.609914 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.610598 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.612650 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:00.613127 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:00.613317 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:00.613846 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:32:00.613955 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:32:00.614033 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:00.614184 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 23:32:00.614256 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:32:00.623982 12702 raft_consensus.cc:1261] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Refusing update from remote peer acd0cc56f42a42f0ad2a448bacc43062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:00.624087 12488 raft_consensus.cc:1261] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Refusing update from remote peer acd0cc56f42a42f0ad2a448bacc43062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:00.624583 13288 consensus_queue.cc:1035] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb6e4a0ea94848b5a2df256fe37536e9" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 39999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:00.624933 13160 consensus_queue.cc:1035] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45affa7eaf354e259778257c2d75662a" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 36737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:32:00.651136 13166 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 23:32:02.237605 12224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:02.237856 12224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:02.241756 12224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3df99f497fc4475b2e702612db07d57: Sending DeleteTablet for 3 replicas of tablet 6f67a27634aa4e4db57d16d8c2e7edff
I20260810 23:32:02.242641 12459 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f67a27634aa4e4db57d16d8c2e7edff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:02 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:34114
I20260810 23:32:02.242654 12903 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f67a27634aa4e4db57d16d8c2e7edff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:02 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:57254
I20260810 23:32:02.242707 12668 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f67a27634aa4e4db57d16d8c2e7edff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:02 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:49942
I20260810 23:32:02.243394 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12347
I20260810 23:32:02.253600 13336 tablet_replica.cc:330] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9: stopping tablet replica
I20260810 23:32:02.253999 13336 raft_consensus.cc:2229] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:02.254369 13336 raft_consensus.cc:2258] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:02.255084 13335 tablet_replica.cc:330] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a: stopping tablet replica
I20260810 23:32:02.255389 13335 raft_consensus.cc:2229] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:02.255661 13335 raft_consensus.cc:2258] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:02.256757 13336 ts_tablet_manager.cc:1905] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:02.265347 13335 ts_tablet_manager.cc:1905] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:02.266033 13334 tablet_replica.cc:330] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062: stopping tablet replica
I20260810 23:32:02.266332 13334 raft_consensus.cc:2229] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:02.266722 13334 raft_consensus.cc:2258] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:02.267658 13334 ts_tablet_manager.cc:1905] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:02.269094 13336 ts_tablet_manager.cc:1918] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:32:02.269289 13336 log.cc:1196] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/6f67a27634aa4e4db57d16d8c2e7edff
I20260810 23:32:02.269865 13336 ts_tablet_manager.cc:1939] T 6f67a27634aa4e4db57d16d8c2e7edff P eb6e4a0ea94848b5a2df256fe37536e9: Deleting consensus metadata
I20260810 23:32:02.271016 12199 catalog_manager.cc:4841] TS eb6e4a0ea94848b5a2df256fe37536e9 (127.9.48.194:39999): tablet 6f67a27634aa4e4db57d16d8c2e7edff (table test-table [id=0af47bba06f4425ca1bcd0d1ddde770d]) successfully deleted
I20260810 23:32:02.273247 13335 ts_tablet_manager.cc:1918] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:32:02.273391 13335 log.cc:1196] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/6f67a27634aa4e4db57d16d8c2e7edff
I20260810 23:32:02.273406 13334 ts_tablet_manager.cc:1918] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:32:02.273552 13334 log.cc:1196] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/6f67a27634aa4e4db57d16d8c2e7edff
I20260810 23:32:02.273891 13335 ts_tablet_manager.cc:1939] T 6f67a27634aa4e4db57d16d8c2e7edff P 45affa7eaf354e259778257c2d75662a: Deleting consensus metadata
I20260810 23:32:02.274423 13334 ts_tablet_manager.cc:1939] T 6f67a27634aa4e4db57d16d8c2e7edff P acd0cc56f42a42f0ad2a448bacc43062: Deleting consensus metadata
I20260810 23:32:02.275063 12198 catalog_manager.cc:4841] TS 45affa7eaf354e259778257c2d75662a (127.9.48.193:36737): tablet 6f67a27634aa4e4db57d16d8c2e7edff (table test-table [id=0af47bba06f4425ca1bcd0d1ddde770d]) successfully deleted
I20260810 23:32:02.275516 12197 catalog_manager.cc:4841] TS acd0cc56f42a42f0ad2a448bacc43062 (127.9.48.195:34805): tablet 6f67a27634aa4e4db57d16d8c2e7edff (table test-table [id=0af47bba06f4425ca1bcd0d1ddde770d]) successfully deleted
Aug 10 23:32:02 dist-test-slave-04bb krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:32:02.286595 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:02.245641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34138 (local address 127.9.48.193:36737)
0810 23:32:02.270163 (+ 24522us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:02.270169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:02.270193 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:32:02.270262 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:02.270268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:02.270353 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:02.270459 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:02.272422 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:02.273614 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:02.275303 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:02.275803 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:02.275941 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:02.280059 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:02.280085 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:02.280142 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:02.280181 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:02.283447 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:02.284103 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:02.284111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:02.284113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:02.284294 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:02.284834 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:02.284843 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:02.284846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:02.285259 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:02.285612 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:02.285961 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 23:32:02.286233 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24383,"thread_start_us":26956,"threads_started":1}
W20260810 23:32:02.287308 12559 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34138
W20260810 23:32:02.287935 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12347: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:32:02.288172 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 12347
I20260810 23:32:02.318423 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12572
Aug 10 23:32:02 dist-test-slave-04bb krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:32:02.334332 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:02.319357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.9.48.194:39999)
0810 23:32:02.319872 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:02.319877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:02.319894 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:32:02.319950 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:02.319954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:02.320013 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:02.320105 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:02.321498 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:02.322272 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:02.323552 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:02.323974 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:02.324071 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:02.328472 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:02.328488 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:02.328502 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:02.328598 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:02.331563 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:02.332296 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:02.332304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:02.332307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:02.332485 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:02.332968 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:02.332972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:02.332974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:02.333357 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:02.333592 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:02.333864 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 23:32:02.334029 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":256,"threads_started":1}
W20260810 23:32:02.335116 12776 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49966
W20260810 23:32:02.335891 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12572: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:32:02.336061 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 12572
I20260810 23:32:02.370868 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12788
Aug 10 23:32:02 dist-test-slave-04bb krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:32:02.386766 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:02.371740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.9.48.195:34805)
0810 23:32:02.372425 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:02.372430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:02.372448 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:32:02.372505 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:02.372509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:02.372648 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:02.372782 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:02.374115 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:02.374878 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:02.376152 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:02.376594 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:02.376722 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:02.381160 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:02.381175 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:02.381192 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:02.381223 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:02.384129 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:02.384969 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:02.384978 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:02.384982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:02.385129 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:02.385477 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:02.385480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:02.385482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:02.385859 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:02.386078 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:02.386316 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 23:32:02.386486 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":273,"threads_started":1}
W20260810 23:32:02.387498 13017 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57282
W20260810 23:32:02.388124 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12788: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:32:02.388276 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 12788
I20260810 23:32:02.420756 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12171
I20260810 23:32:02.421957 12286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:02.541468 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 12171
2026-08-10T23:32:02Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9312 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:32:02 dist-test-slave-04bb krb5kdc[13347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:32:02 dist-test-slave-04bb krb5kdc[13347](info): set up 2 sockets
Aug 10 23:32:02 dist-test-slave-04bb krb5kdc[13347](info): commencing operation
krb5kdc: starting...
W20260810 23:32:03.859177 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:32:04 dist-test-slave-04bb krb5kdc[13347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:32:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:32:04.076793 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:41407
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:41407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:04.418702 13369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:04.419027 13369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:04.419188 13369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:04.426379 13369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:04.426527 13369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:04.426628 13369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:04.426733 13369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:04.426810 13369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:04.436259 13369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:41407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:41407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:04.437544 13369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:04.438872 13369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:04.450770 13380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:04.450896 13385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:04.450798 13382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:04.451761 13369 server_base.cc:1034] running on GCE node
I20260810 23:32:04.452494 13369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:04.453917 13369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:04.455120 13369 hybrid_clock.cc:648] HybridClock initialized: now 1786404724455080 us; error 32 us; skew 500 ppm
I20260810 23:32:04.455475 13369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:04 dist-test-slave-04bb krb5kdc[13347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404724, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:04.458562 13369 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:32:04.459995 13369 webserver.cc:460] Webserver started at http://127.9.48.254:41941/ using document root <none> and password file <none>
I20260810 23:32:04.460749 13369 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:04.460894 13369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:04.461215 13369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:04.464128 13369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "73116f0015c14d91be8fb8df8ca00f23"
format_stamp: "Formatted at 2026-08-10 23:32:04 on dist-test-slave-04bb"
server_key: "a6af2f36f236e7f96b984604c9e23a7d"
server_key_iv: "17ea3c0b7587b98f9b424273ba703a9f"
server_key_version: "encryptionkey@0"
I20260810 23:32:04.464986 13369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "73116f0015c14d91be8fb8df8ca00f23"
format_stamp: "Formatted at 2026-08-10 23:32:04 on dist-test-slave-04bb"
server_key: "a6af2f36f236e7f96b984604c9e23a7d"
server_key_iv: "17ea3c0b7587b98f9b424273ba703a9f"
server_key_version: "encryptionkey@0"
I20260810 23:32:04.469985 13369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 23:32:04.473304 13398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:04.474560 13369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:32:04.474750 13369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "73116f0015c14d91be8fb8df8ca00f23"
format_stamp: "Formatted at 2026-08-10 23:32:04 on dist-test-slave-04bb"
server_key: "a6af2f36f236e7f96b984604c9e23a7d"
server_key_iv: "17ea3c0b7587b98f9b424273ba703a9f"
server_key_version: "encryptionkey@0"
I20260810 23:32:04.474951 13369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:04.493018 13369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:04.496992 13369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:04.497295 13369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:04.515852 13369 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:41407
I20260810 23:32:04.515898 13492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:41407 every 8 connection(s)
I20260810 23:32:04.517284 13369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:04.521023 13494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:04.526036 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 13369
I20260810 23:32:04.526314 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:32:04.527114 9411 external_mini_cluster.cc:1428] Setting key 8c85051cd81ccdd341b26c2ee3c81057
I20260810 23:32:04.532624 13494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23: Bootstrap starting.
I20260810 23:32:04.536268 13494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:04.537556 13494 log.cc:824] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:04.541028 13494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23: No bootstrap required, opened a new log
Aug 10 23:32:04 dist-test-slave-04bb krb5kdc[13347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:04.548764 13494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73116f0015c14d91be8fb8df8ca00f23" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41407 } }
I20260810 23:32:04.549142 13494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:04.549265 13494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73116f0015c14d91be8fb8df8ca00f23, State: Initialized, Role: FOLLOWER
I20260810 23:32:04.549857 13494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73116f0015c14d91be8fb8df8ca00f23" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41407 } }
I20260810 23:32:04.550124 13494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:32:04.550277 13494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:32:04.550470 13494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:04.550851 13504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:04.529253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38516 (local address 127.9.48.254:41407)
0810 23:32:04.530093 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:04.530105 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:04.530150 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:32:04.531842 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:04.531862 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:04.532224 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:04.532597 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:04.534349 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:04.535628 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:04.537536 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:04.538538 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:04.538758 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:04.543269 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:04.543277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:04.543289 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:04.543331 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:04.546459 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:04.547171 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:04.547177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:04.547183 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:04.547353 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:04.547762 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:04.547766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:04.547768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:04.548281 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:04.548624 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:04.549045 (+ 421us) server_negotiation.cc:299] Negotiation successful
0810 23:32:04.549675 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":401,"threads_started":1}
I20260810 23:32:04.552861 13494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73116f0015c14d91be8fb8df8ca00f23" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41407 } }
I20260810 23:32:04.553359 13494 leader_election.cc:302] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73116f0015c14d91be8fb8df8ca00f23; no voters:
I20260810 23:32:04.553957 13494 leader_election.cc:288] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:32:04.554059 13506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:04.554840 13506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [term 1 LEADER]: Becoming Leader. State: Replica: 73116f0015c14d91be8fb8df8ca00f23, State: Running, Role: LEADER
I20260810 23:32:04.555397 13506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73116f0015c14d91be8fb8df8ca00f23" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41407 } }
I20260810 23:32:04.556097 13494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:04.558070 13511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73116f0015c14d91be8fb8df8ca00f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73116f0015c14d91be8fb8df8ca00f23" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41407 } } }
I20260810 23:32:04.558480 13511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:04.561412 13512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73116f0015c14d91be8fb8df8ca00f23. Latest consensus state: current_term: 1 leader_uuid: "73116f0015c14d91be8fb8df8ca00f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73116f0015c14d91be8fb8df8ca00f23" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41407 } } }
I20260810 23:32:04.561776 13512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:04.562089 13517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:04.568485 13517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:04.586539 13517 catalog_manager.cc:1347] Generated new cluster ID: 2cb8ab9dcb174ac1aead1f8175ca765b
I20260810 23:32:04.586664 13517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:04.615756 13517 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:32:04.617543 13517 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:04.625671 13517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73116f0015c14d91be8fb8df8ca00f23: Generated new TSK 0
I20260810 23:32:04.626538 13517 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
W20260810 23:32:05.720043 9411 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.9.48.193: real 1.086s user 0.000s sys 0.016s
I20260810 23:32:05.721190 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41407
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:06.077481 13543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:06.077792 13543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:06.077916 13543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:06.085414 13543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:06.085517 13543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:06.085745 13543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:32:06.099468 13543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:06.101156 13543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:06.102708 13543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:06.117300 13549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:06.117277 13551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:06.117197 13555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:06.118237 13543 server_base.cc:1034] running on GCE node
I20260810 23:32:06.118906 13543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:06.119822 13543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:06.121037 13543 hybrid_clock.cc:648] HybridClock initialized: now 1786404726120993 us; error 32 us; skew 500 ppm
I20260810 23:32:06.121376 13543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:06 dist-test-slave-04bb krb5kdc[13347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404726, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:06.128193 13543 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:32:06.129902 13543 webserver.cc:460] Webserver started at http://127.9.48.193:35215/ using document root <none> and password file <none>
I20260810 23:32:06.130610 13543 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:06.130766 13543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:06.131142 13543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:06.134310 13543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "dcc409baa99442309b6bdec6a413577e"
format_stamp: "Formatted at 2026-08-10 23:32:06 on dist-test-slave-04bb"
server_key: "e9368993ebaf69724e6a2785c09b3bd0"
server_key_iv: "de1248458885c2dadaa8ac5fd49ba459"
server_key_version: "encryptionkey@0"
I20260810 23:32:06.135277 13543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "dcc409baa99442309b6bdec6a413577e"
format_stamp: "Formatted at 2026-08-10 23:32:06 on dist-test-slave-04bb"
server_key: "e9368993ebaf69724e6a2785c09b3bd0"
server_key_iv: "de1248458885c2dadaa8ac5fd49ba459"
server_key_version: "encryptionkey@0"
I20260810 23:32:06.140758 13543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260810 23:32:06.144488 13565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:06.145967 13543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:06.146144 13543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "dcc409baa99442309b6bdec6a413577e"
format_stamp: "Formatted at 2026-08-10 23:32:06 on dist-test-slave-04bb"
server_key: "e9368993ebaf69724e6a2785c09b3bd0"
server_key_iv: "de1248458885c2dadaa8ac5fd49ba459"
server_key_version: "encryptionkey@0"
I20260810 23:32:06.146373 13543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:06.183112 13543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:06.187719 13543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:06.188041 13543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:06.189118 13543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:06.190575 13543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:06.190692 13543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:06.190850 13543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:06.190932 13543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:06.229090 13543 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:35171
I20260810 23:32:06.229141 13736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:35171 every 8 connection(s)
I20260810 23:32:06.230435 13543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:32:06.233943 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 13543
I20260810 23:32:06.234172 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:32:06.235092 9411 external_mini_cluster.cc:1428] Setting key c31ca3b9c185435864400dafeab111fa
Aug 10 23:32:06 dist-test-slave-04bb krb5kdc[13347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404726, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:06.252478 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:06.233206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:38299 (local address 127.9.48.254:41407)
0810 23:32:06.233689 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:06.233697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:06.234949 (+ 1252us) server_negotiation.cc:407] Connection header received
0810 23:32:06.237271 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:06.237277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:06.237366 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:06.237484 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:06.239066 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:06.240284 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:06.241935 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:06.242650 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:06.242794 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:06.246407 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:06.246413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:06.246416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:06.246448 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:06.249160 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:06.249964 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:06.249969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:06.249971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:06.250137 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:06.250613 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:06.250618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:06.250622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:06.250923 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:06.251084 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:06.251814 (+ 730us) server_negotiation.cc:299] Negotiation successful
0810 23:32:06.252045 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":253,"threads_started":1}
I20260810 23:32:06.253554 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:06.233847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:41407 (local address 127.9.48.193:38299)
0810 23:32:06.234730 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:06.234806 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:06.236952 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:06.237726 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:06.237735 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:06.238298 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:06.238866 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:06.238885 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:06.240457 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:06.240461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:06.241787 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:06.241800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:06.242868 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:06.242873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:06.243119 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:06.243838 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:06.243866 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:06.246182 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:06.249304 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:06.249314 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:06.249330 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:06.249828 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:06.250283 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:06.250289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:06.250293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:06.250489 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:06.251057 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:06.251065 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:06.251449 (+ 384us) client_negotiation.cc:769] Sending connection context
0810 23:32:06.251723 (+ 274us) client_negotiation.cc:240] Negotiation successful
0810 23:32:06.252292 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":342,"threads_started":1}
I20260810 23:32:06.255187 13737 heartbeater.cc:347] Connected to a master server at 127.9.48.254:41407
I20260810 23:32:06.255535 13737 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:06.256389 13737 heartbeater.cc:511] Master 127.9.48.254:41407 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
I20260810 23:32:06.258663 13431 ts_manager.cc:194] Registered new tserver with Master: dcc409baa99442309b6bdec6a413577e (127.9.48.193:35171)
I20260810 23:32:06.260106 13431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:38299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
W20260810 23:32:06.475411 9411 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.9.48.194: real 0.237s user 0.004s sys 0.011s
I20260810 23:32:06.476159 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41407
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:06.807812 13748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:06.808151 13748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:06.808333 13748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:06.815431 13748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:06.815578 13748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:06.815769 13748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:32:06.825259 13748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:06.826517 13748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:06.827854 13748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:06.841501 13758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:06.841647 13755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:06.841655 13753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:06.842525 13748 server_base.cc:1034] running on GCE node
I20260810 23:32:06.843138 13748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:06.844007 13748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:06.845202 13748 hybrid_clock.cc:648] HybridClock initialized: now 1786404726845153 us; error 44 us; skew 500 ppm
I20260810 23:32:06.845465 13748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:06 dist-test-slave-04bb krb5kdc[13347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404726, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:06.848691 13748 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:32:06.850144 13748 webserver.cc:460] Webserver started at http://127.9.48.194:38573/ using document root <none> and password file <none>
I20260810 23:32:06.850880 13748 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:06.851016 13748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:06.851389 13748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:06.854357 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "20faaa0d35814dfd87be78dd42a0a206"
format_stamp: "Formatted at 2026-08-10 23:32:06 on dist-test-slave-04bb"
server_key: "4bad1a59fb1dbe7953c71e0d5f95d425"
server_key_iv: "6fea02ac9cda11750f793ce69ab92f2c"
server_key_version: "encryptionkey@0"
I20260810 23:32:06.855144 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "20faaa0d35814dfd87be78dd42a0a206"
format_stamp: "Formatted at 2026-08-10 23:32:06 on dist-test-slave-04bb"
server_key: "4bad1a59fb1dbe7953c71e0d5f95d425"
server_key_iv: "6fea02ac9cda11750f793ce69ab92f2c"
server_key_version: "encryptionkey@0"
I20260810 23:32:06.860447 13748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 23:32:06.863926 13774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:06.865279 13748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:06.865474 13748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "20faaa0d35814dfd87be78dd42a0a206"
format_stamp: "Formatted at 2026-08-10 23:32:06 on dist-test-slave-04bb"
server_key: "4bad1a59fb1dbe7953c71e0d5f95d425"
server_key_iv: "6fea02ac9cda11750f793ce69ab92f2c"
server_key_version: "encryptionkey@0"
I20260810 23:32:06.865682 13748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:06.886108 13748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:06.890483 13748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:06.890774 13748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:06.891767 13748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:06.893168 13748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:06.893280 13748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:06.893438 13748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:06.893514 13748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:06.928283 13748 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:34793
I20260810 23:32:06.928380 13954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:34793 every 8 connection(s)
I20260810 23:32:06.929630 13748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:32:06.936311 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 13748
I20260810 23:32:06.936534 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:32:06.937062 9411 external_mini_cluster.cc:1428] Setting key 61873073d137945379ed342775bffe0f
Aug 10 23:32:06 dist-test-slave-04bb krb5kdc[13347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404726, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:06.952113 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:06.932345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:48903 (local address 127.9.48.254:41407)
0810 23:32:06.932880 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:06.932885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:06.933760 (+ 875us) server_negotiation.cc:407] Connection header received
0810 23:32:06.935781 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:06.935789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:06.935951 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:06.936097 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:06.937886 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:06.938907 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:06.940490 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:06.941359 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:06.941529 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:06.945227 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:06.945237 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:06.945246 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:06.945296 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:06.948460 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:06.949512 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:06.949517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:06.949519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:06.949661 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:06.950253 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:06.950258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:06.950264 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:06.950553 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:06.950695 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:06.951642 (+ 947us) server_negotiation.cc:299] Negotiation successful
0810 23:32:06.951827 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":295,"threads_started":1}
I20260810 23:32:06.953644 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:06.932819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:41407 (local address 127.9.48.194:48903)
0810 23:32:06.933584 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:06.933629 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:06.935465 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:06.936320 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:06.936328 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:06.937001 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:06.937659 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:06.937686 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:06.939088 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:06.939093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:06.940328 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:06.940341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:06.941580 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:06.941585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:06.941780 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:06.942489 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:06.942515 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:06.944983 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:06.948682 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:06.948693 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:06.948711 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:06.949350 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:06.949801 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:06.949816 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:06.949823 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:06.950090 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:06.950699 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:06.950710 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:06.951172 (+ 462us) client_negotiation.cc:769] Sending connection context
0810 23:32:06.951516 (+ 344us) client_negotiation.cc:240] Negotiation successful
0810 23:32:06.952086 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":295,"threads_started":1}
I20260810 23:32:06.955219 13955 heartbeater.cc:347] Connected to a master server at 127.9.48.254:41407
I20260810 23:32:06.955582 13955 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:06.956493 13955 heartbeater.cc:511] Master 127.9.48.254:41407 requested a full tablet report, sending...
I20260810 23:32:06.958330 13431 ts_manager.cc:194] Registered new tserver with Master: 20faaa0d35814dfd87be78dd42a0a206 (127.9.48.194:34793)
I20260810 23:32:06.959630 13431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:48903
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:32:06.992380 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41407
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:32:07.263087 13737 heartbeater.cc:503] Master 127.9.48.254:41407 was elected leader, sending a full tablet report...
W20260810 23:32:07.335685 13963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:07.335968 13963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:07.336064 13963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:07.343231 13963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:07.343334 13963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:07.343442 13963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:32:07.353264 13963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:07.354478 13963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:07.355741 13963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:07.370999 13974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:07.371130 13976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:07.371816 13979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:07.373519 13963 server_base.cc:1034] running on GCE node
I20260810 23:32:07.374396 13963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:07.375697 13963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:07.376922 13963 hybrid_clock.cc:648] HybridClock initialized: now 1786404727376855 us; error 56 us; skew 500 ppm
I20260810 23:32:07.377290 13963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:07 dist-test-slave-04bb krb5kdc[13347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404727, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:07.381518 13963 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:32:07.383623 13963 webserver.cc:460] Webserver started at http://127.9.48.195:38183/ using document root <none> and password file <none>
I20260810 23:32:07.384742 13963 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:07.384899 13963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:07.385293 13963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:07.389734 13963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "022dd4a01e944a8dbc0124d24a46c7fd"
format_stamp: "Formatted at 2026-08-10 23:32:07 on dist-test-slave-04bb"
server_key: "74ecb14610230fc988f64a10b8725a96"
server_key_iv: "f16179798fc0e98fd659c9adecab8633"
server_key_version: "encryptionkey@0"
I20260810 23:32:07.390872 13963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "022dd4a01e944a8dbc0124d24a46c7fd"
format_stamp: "Formatted at 2026-08-10 23:32:07 on dist-test-slave-04bb"
server_key: "74ecb14610230fc988f64a10b8725a96"
server_key_iv: "f16179798fc0e98fd659c9adecab8633"
server_key_version: "encryptionkey@0"
I20260810 23:32:07.399158 13963 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260810 23:32:07.403514 13988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:07.405283 13963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:32:07.405472 13963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "022dd4a01e944a8dbc0124d24a46c7fd"
format_stamp: "Formatted at 2026-08-10 23:32:07 on dist-test-slave-04bb"
server_key: "74ecb14610230fc988f64a10b8725a96"
server_key_iv: "f16179798fc0e98fd659c9adecab8633"
server_key_version: "encryptionkey@0"
I20260810 23:32:07.405686 13963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:07.429060 13963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:07.433590 13963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:07.433890 13963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:07.434834 13963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:07.436282 13963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:07.436384 13963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:07.436570 13963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:07.436664 13963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260810 23:32:07.471225 13963 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:44545
I20260810 23:32:07.471311 14179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:44545 every 8 connection(s)
I20260810 23:32:07.472497 13963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:32:07.473069 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 13963
I20260810 23:32:07.473312 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:32:07.473878 9411 external_mini_cluster.cc:1428] Setting key 5ec69b6c3a0925e3a2dc603a925870bc
Aug 10 23:32:07 dist-test-slave-04bb krb5kdc[13347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404727, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:07.493571 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:07.475382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:36971 (local address 127.9.48.254:41407)
0810 23:32:07.476164 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:07.476168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:07.478198 (+ 2030us) server_negotiation.cc:407] Connection header received
0810 23:32:07.479202 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:07.479226 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:07.479299 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:07.479427 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:07.480931 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:07.481805 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:07.483456 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:07.483875 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:07.483979 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:07.487331 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:07.487336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:07.487340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:07.487373 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:07.490788 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:07.491456 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:07.491460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:07.491462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:07.491592 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:07.491980 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:07.491983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:07.491986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:07.492323 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:07.492445 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:07.493156 (+ 711us) server_negotiation.cc:299] Negotiation successful
0810 23:32:07.493314 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":545,"threads_started":1}
I20260810 23:32:07.494493 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:07.475986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:41407 (local address 127.9.48.195:36971)
0810 23:32:07.477035 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:07.477087 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:07.478959 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:07.479638 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:07.479653 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:07.480168 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:07.480775 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:07.480793 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:07.481949 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:07.481953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:07.483301 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:07.483313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:07.484059 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:07.484063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:07.484264 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:07.484965 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:07.484986 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:07.487120 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:07.490912 (+ 3792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:07.490918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:07.490929 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:07.491321 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:07.491696 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:07.491699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:07.491702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:07.491868 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:07.492431 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:07.492436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:07.492785 (+ 349us) client_negotiation.cc:769] Sending connection context
0810 23:32:07.493006 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 23:32:07.493353 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":520,"threads_started":1}
I20260810 23:32:07.495932 14180 heartbeater.cc:347] Connected to a master server at 127.9.48.254:41407
I20260810 23:32:07.496197 14180 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:07.496932 14180 heartbeater.cc:511] Master 127.9.48.254:41407 requested a full tablet report, sending...
I20260810 23:32:07.498345 13431 ts_manager.cc:194] Registered new tserver with Master: 022dd4a01e944a8dbc0124d24a46c7fd (127.9.48.195:44545)
I20260810 23:32:07.499249 13431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:36971
I20260810 23:32:07.502206 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:32:07.542204 14183 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:07.539579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.9.48.254:41407)
0810 23:32:07.539774 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:07.539779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:07.540306 (+ 527us) server_negotiation.cc:407] Connection header received
0810 23:32:07.540459 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:07.540464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:07.540591 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:07.540693 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:07.541851 (+ 1158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38518: BlockingRecv error: recv got EOF from 127.0.0.1:38518 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:32:07.545768 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 13543
W20260810 23:32:07.548555 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 13543: Not authorized: Client connection negotiation failed: client connection to 127.9.48.193:35171: server requires authentication, but client does not have Kerberos credentials available
I20260810 23:32:07.548758 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 13543
I20260810 23:32:07.576140 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 13748
W20260810 23:32:07.579499 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 13748: Not authorized: Client connection negotiation failed: client connection to 127.9.48.194:34793: server requires authentication, but client does not have Kerberos credentials available
I20260810 23:32:07.579672 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 13748
I20260810 23:32:07.603030 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 13963
W20260810 23:32:07.606076 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 13963: Not authorized: Client connection negotiation failed: client connection to 127.9.48.195:44545: server requires authentication, but client does not have Kerberos credentials available
I20260810 23:32:07.606194 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 13963
I20260810 23:32:07.630817 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 13369
I20260810 23:32:07.632237 13486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:07.738960 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 13369
2026-08-10T23:32:07Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5194 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:32:07 dist-test-slave-04bb krb5kdc[14204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:32:07 dist-test-slave-04bb krb5kdc[14204](info): set up 2 sockets
Aug 10 23:32:07 dist-test-slave-04bb krb5kdc[14204](info): commencing operation
krb5kdc: starting...
W20260810 23:32:09.105067 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.315s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:32:09 dist-test-slave-04bb krb5kdc[14204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:32:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:32:09.328118 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:38121
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:41825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:38121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:09.660130 14223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:09.660495 14223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:09.660693 14223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:09.668063 14223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:09.668221 14223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:09.668325 14223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:09.668428 14223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:09.668512 14223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:09.678568 14223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:41825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:38121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:38121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:09.679920 14223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:09.681386 14223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:09.693912 14235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:09.694293 14232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:09.694684 14230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:09.695096 14223 server_base.cc:1034] running on GCE node
I20260810 23:32:09.696002 14223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:09.697561 14223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:09.698788 14223 hybrid_clock.cc:648] HybridClock initialized: now 1786404729698730 us; error 58 us; skew 500 ppm
I20260810 23:32:09.699150 14223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:09 dist-test-slave-04bb krb5kdc[14204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404729, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:09.702400 14223 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:32:09.703877 14223 webserver.cc:460] Webserver started at http://127.9.48.254:39757/ using document root <none> and password file <none>
I20260810 23:32:09.704641 14223 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:09.704788 14223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:09.705107 14223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:09.708252 14223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "e1b702c2f5914e7bb8f3fc40f7aab9e4"
format_stamp: "Formatted at 2026-08-10 23:32:09 on dist-test-slave-04bb"
server_key: "4482a26112896765a223c4b159b70558"
server_key_iv: "3179805e296615fe31687fc254df8b4f"
server_key_version: "encryptionkey@0"
I20260810 23:32:09.709124 14223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "e1b702c2f5914e7bb8f3fc40f7aab9e4"
format_stamp: "Formatted at 2026-08-10 23:32:09 on dist-test-slave-04bb"
server_key: "4482a26112896765a223c4b159b70558"
server_key_iv: "3179805e296615fe31687fc254df8b4f"
server_key_version: "encryptionkey@0"
I20260810 23:32:09.714344 14223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:32:09.717850 14248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:09.719169 14223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:32:09.719362 14223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "e1b702c2f5914e7bb8f3fc40f7aab9e4"
format_stamp: "Formatted at 2026-08-10 23:32:09 on dist-test-slave-04bb"
server_key: "4482a26112896765a223c4b159b70558"
server_key_iv: "3179805e296615fe31687fc254df8b4f"
server_key_version: "encryptionkey@0"
I20260810 23:32:09.719559 14223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:09.776778 14223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:09.781279 14223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:09.781576 14223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:09.800983 14223 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:38121
I20260810 23:32:09.801017 14348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:38121 every 8 connection(s)
I20260810 23:32:09.802366 14223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:09.806317 14349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:09.808930 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 14223
I20260810 23:32:09.809171 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:32:09.810088 9411 external_mini_cluster.cc:1428] Setting key 6ea8884b38a34d4f8809ee9b739d2f72
I20260810 23:32:09.819993 14349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4: Bootstrap starting.
I20260810 23:32:09.824064 14349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4: Neither blocks nor log segments found. Creating new log.
Aug 10 23:32:09 dist-test-slave-04bb krb5kdc[14204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:09.825317 14349 log.cc:824] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:09.828352 14349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4: No bootstrap required, opened a new log
I20260810 23:32:09.833222 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:09.812141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.9.48.254:38121)
0810 23:32:09.813022 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:09.813034 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:09.813090 (+ 56us) server_negotiation.cc:407] Connection header received
0810 23:32:09.814734 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:09.814756 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:09.815133 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:09.815439 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:09.817460 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:09.818768 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:09.820696 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:09.821399 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:09.821546 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:09.825905 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:09.825912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:09.825929 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:09.825964 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:09.829089 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:09.829800 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:09.829808 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:09.829814 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:09.829989 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:09.830388 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:09.830392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:09.830417 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:09.830986 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:09.831330 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:09.831785 (+ 455us) server_negotiation.cc:299] Negotiation successful
0810 23:32:09.832185 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":441,"threads_started":1}
I20260810 23:32:09.837047 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:32:09.837248 14349 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b702c2f5914e7bb8f3fc40f7aab9e4" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38121 } }
I20260810 23:32:09.837621 14349 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:09.837741 14349 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1b702c2f5914e7bb8f3fc40f7aab9e4, State: Initialized, Role: FOLLOWER
I20260810 23:32:09.838292 14349 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b702c2f5914e7bb8f3fc40f7aab9e4" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38121 } }
I20260810 23:32:09.838615 14349 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:32:09.838775 14349 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:32:09.838965 14349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:09.841291 14349 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b702c2f5914e7bb8f3fc40f7aab9e4" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38121 } }
I20260810 23:32:09.841821 14349 leader_election.cc:302] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1b702c2f5914e7bb8f3fc40f7aab9e4; no voters:
I20260810 23:32:09.842541 14349 leader_election.cc:288] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:32:09.842655 14362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:09.844022 14362 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [term 1 LEADER]: Becoming Leader. State: Replica: e1b702c2f5914e7bb8f3fc40f7aab9e4, State: Running, Role: LEADER
I20260810 23:32:09.844645 14362 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b702c2f5914e7bb8f3fc40f7aab9e4" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38121 } }
I20260810 23:32:09.845083 14349 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:09.848346 14369 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1b702c2f5914e7bb8f3fc40f7aab9e4. Latest consensus state: current_term: 1 leader_uuid: "e1b702c2f5914e7bb8f3fc40f7aab9e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b702c2f5914e7bb8f3fc40f7aab9e4" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38121 } } }
I20260810 23:32:09.848847 14369 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:09.849416 14366 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1b702c2f5914e7bb8f3fc40f7aab9e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1b702c2f5914e7bb8f3fc40f7aab9e4" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38121 } } }
I20260810 23:32:09.849774 14366 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:09.861948 14381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:09.868916 14381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:09.887435 14381 catalog_manager.cc:1347] Generated new cluster ID: 280a8051225a44cbbb7a242610bb0589
I20260810 23:32:09.887562 14381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:09.906642 14381 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:32:09.908483 14381 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:09.915736 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:09.907278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.9.48.254:38121)
0810 23:32:09.907491 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:09.907498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:09.908241 (+ 743us) server_negotiation.cc:407] Connection header received
0810 23:32:09.908475 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:09.908482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:09.908628 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:09.908753 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:09.909928 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:09.910720 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:09.912083 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:09.912647 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:09.912803 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:09.914087 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:09.914094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:09.914097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:09.914812 (+ 715us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:09.914886 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:09.914911 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:09.915044 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:09.915201 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 23:32:09.915309 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:32:09.918727 14381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1b702c2f5914e7bb8f3fc40f7aab9e4: Generated new TSK 0
I20260810 23:32:09.919842 14381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:32:09.932364 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 14223
I20260810 23:32:09.933498 14341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:10.023432 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 14223
2026-08-10T23:32:10Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2281 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:32:10 dist-test-slave-04bb krb5kdc[14420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:32:10 dist-test-slave-04bb krb5kdc[14420](info): set up 2 sockets
Aug 10 23:32:10 dist-test-slave-04bb krb5kdc[14420](info): commencing operation
krb5kdc: starting...
W20260810 23:32:11.336834 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:32:11 dist-test-slave-04bb krb5kdc[14420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:32:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:32:11.561509 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:38987
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:36339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:38987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:11.889854 14448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:11.890223 14448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:11.890394 14448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:11.897655 14448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:11.897804 14448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:11.897917 14448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:11.898022 14448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:11.898119 14448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:11.907894 14448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:36339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:38987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:38987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:11.909310 14448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:11.910657 14448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:11.923584 14462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:11.923607 14459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:11.923611 14456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:11.924471 14448 server_base.cc:1034] running on GCE node
I20260810 23:32:11.925338 14448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:11.926761 14448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:11.927992 14448 hybrid_clock.cc:648] HybridClock initialized: now 1786404731927945 us; error 30 us; skew 500 ppm
I20260810 23:32:11.928396 14448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:11 dist-test-slave-04bb krb5kdc[14420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404731, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:11.937434 14448 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:32:11.939072 14448 webserver.cc:460] Webserver started at http://127.9.48.254:40897/ using document root <none> and password file <none>
I20260810 23:32:11.939821 14448 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:11.939977 14448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:11.940323 14448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:11.943472 14448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "c4bcfa3d9704475c9b64d2d6536ba183"
format_stamp: "Formatted at 2026-08-10 23:32:11 on dist-test-slave-04bb"
server_key: "6abad3b48985e30145799b26cc091926"
server_key_iv: "c8cef69da3200724c22cc85850c42f34"
server_key_version: "encryptionkey@0"
I20260810 23:32:11.944272 14448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "c4bcfa3d9704475c9b64d2d6536ba183"
format_stamp: "Formatted at 2026-08-10 23:32:11 on dist-test-slave-04bb"
server_key: "6abad3b48985e30145799b26cc091926"
server_key_iv: "c8cef69da3200724c22cc85850c42f34"
server_key_version: "encryptionkey@0"
I20260810 23:32:11.949753 14448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:32:11.953285 14476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:11.954599 14448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:32:11.954804 14448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "c4bcfa3d9704475c9b64d2d6536ba183"
format_stamp: "Formatted at 2026-08-10 23:32:11 on dist-test-slave-04bb"
server_key: "6abad3b48985e30145799b26cc091926"
server_key_iv: "c8cef69da3200724c22cc85850c42f34"
server_key_version: "encryptionkey@0"
I20260810 23:32:11.955006 14448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:11.970924 14448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:11.975481 14448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:11.975788 14448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:11.994777 14448 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:38987
I20260810 23:32:11.994810 14573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:38987 every 8 connection(s)
I20260810 23:32:11.996145 14448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:11.999904 14575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:12.004161 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 14448
I20260810 23:32:12.004412 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:32:12.005098 9411 external_mini_cluster.cc:1428] Setting key 4090f99ea3afc92b6f53b10ce623330c
I20260810 23:32:12.011075 14575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183: Bootstrap starting.
I20260810 23:32:12.014740 14575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:12.016022 14575 log.cc:824] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:12.019794 14575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183: No bootstrap required, opened a new log
Aug 10 23:32:12 dist-test-slave-04bb krb5kdc[14420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:12.027565 14575 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bcfa3d9704475c9b64d2d6536ba183" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38987 } }
I20260810 23:32:12.027992 14575 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:12.028110 14575 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4bcfa3d9704475c9b64d2d6536ba183, State: Initialized, Role: FOLLOWER
I20260810 23:32:12.028810 14575 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bcfa3d9704475c9b64d2d6536ba183" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38987 } }
I20260810 23:32:12.029086 14575 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:32:12.029243 14575 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:32:12.029436 14575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:12.029723 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:12.007137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47582 (local address 127.9.48.254:38987)
0810 23:32:12.007990 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:12.008002 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:12.008051 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:32:12.009825 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:12.009847 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:12.010224 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:12.010532 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:12.012400 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:12.014000 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:12.015846 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:12.016829 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:12.017115 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:12.021846 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:12.021854 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:12.021872 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:12.021916 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:12.025311 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:12.026027 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:12.026037 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:12.026043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:12.026222 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:12.026662 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:12.026666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:12.026669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:12.027189 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:12.027511 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:12.027995 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 23:32:12.028428 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":377,"threads_started":1}
I20260810 23:32:12.031847 14575 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bcfa3d9704475c9b64d2d6536ba183" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38987 } }
I20260810 23:32:12.032415 14575 leader_election.cc:302] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4bcfa3d9704475c9b64d2d6536ba183; no voters:
I20260810 23:32:12.033026 14575 leader_election.cc:288] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:32:12.033166 14585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:12.033946 14585 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [term 1 LEADER]: Becoming Leader. State: Replica: c4bcfa3d9704475c9b64d2d6536ba183, State: Running, Role: LEADER
I20260810 23:32:12.034482 14585 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bcfa3d9704475c9b64d2d6536ba183" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38987 } }
I20260810 23:32:12.035194 14575 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:12.037871 14586 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4bcfa3d9704475c9b64d2d6536ba183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bcfa3d9704475c9b64d2d6536ba183" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38987 } } }
I20260810 23:32:12.038240 14586 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:12.037915 14588 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4bcfa3d9704475c9b64d2d6536ba183. Latest consensus state: current_term: 1 leader_uuid: "c4bcfa3d9704475c9b64d2d6536ba183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4bcfa3d9704475c9b64d2d6536ba183" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38987 } } }
I20260810 23:32:12.038422 14588 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:12.040673 14592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:12.047770 14592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:12.058395 14592 catalog_manager.cc:1347] Generated new cluster ID: 4c98cdd2f5ab44109113be5cc20a7a0a
I20260810 23:32:12.058519 14592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:12.099269 14592 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:32:12.101521 14592 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:12.118398 14592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4bcfa3d9704475c9b64d2d6536ba183: Generated new TSK 0
I20260810 23:32:12.119481 14592 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:32:12.180852 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:38987
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:36339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:12.518761 14621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:12.519066 14621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:12.519243 14621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:12.526300 14621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:12.526448 14621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:12.526620 14621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:32:12.536100 14621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:36339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:38987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:12.537533 14621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:12.539100 14621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:12.552551 14632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:12.552600 14630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:12.552897 14629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:12.553699 14621 server_base.cc:1034] running on GCE node
I20260810 23:32:12.554382 14621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:12.555317 14621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:12.556655 14621 hybrid_clock.cc:648] HybridClock initialized: now 1786404732556594 us; error 58 us; skew 500 ppm
I20260810 23:32:12.556988 14621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:12 dist-test-slave-04bb krb5kdc[14420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404732, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:12.560191 14621 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:32:12.561712 14621 webserver.cc:460] Webserver started at http://127.9.48.193:39909/ using document root <none> and password file <none>
I20260810 23:32:12.562399 14621 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:12.562551 14621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:12.562858 14621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:12.565810 14621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "34a5c14e39574e4484c965c7e54dd711"
format_stamp: "Formatted at 2026-08-10 23:32:12 on dist-test-slave-04bb"
server_key: "a87e4016cc9b243ae6b74dc9fd9eb6c6"
server_key_iv: "4ebb85a983bb828dd40acb236db5a791"
server_key_version: "encryptionkey@0"
I20260810 23:32:12.566597 14621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "34a5c14e39574e4484c965c7e54dd711"
format_stamp: "Formatted at 2026-08-10 23:32:12 on dist-test-slave-04bb"
server_key: "a87e4016cc9b243ae6b74dc9fd9eb6c6"
server_key_iv: "4ebb85a983bb828dd40acb236db5a791"
server_key_version: "encryptionkey@0"
I20260810 23:32:12.571800 14621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 23:32:12.575317 14641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:12.576740 14621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:12.576921 14621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "34a5c14e39574e4484c965c7e54dd711"
format_stamp: "Formatted at 2026-08-10 23:32:12 on dist-test-slave-04bb"
server_key: "a87e4016cc9b243ae6b74dc9fd9eb6c6"
server_key_iv: "4ebb85a983bb828dd40acb236db5a791"
server_key_version: "encryptionkey@0"
I20260810 23:32:12.577133 14621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:12.599413 14621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:12.603477 14621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:12.603775 14621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:12.604789 14621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:12.606096 14621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:12.606209 14621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:12.606364 14621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:12.606439 14621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:12.642059 14621 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:35313
I20260810 23:32:12.642095 14819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:35313 every 8 connection(s)
I20260810 23:32:12.643472 14621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:32:12.645318 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 14621
I20260810 23:32:12.645542 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:32:12.646456 9411 external_mini_cluster.cc:1428] Setting key 82546a3ce6b10e10cc9d67e3d7b49cec
Aug 10 23:32:12 dist-test-slave-04bb krb5kdc[14420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404732, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:12.665436 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:12.646246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:49985 (local address 127.9.48.254:38987)
0810 23:32:12.646757 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:12.646762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:12.648159 (+ 1397us) server_negotiation.cc:407] Connection header received
0810 23:32:12.650206 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:12.650214 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:12.650322 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:12.650448 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:12.651888 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:12.653069 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:12.654606 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:12.655337 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:12.655487 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:12.659563 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:12.659569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:12.659572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:12.659603 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:12.662282 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:12.663016 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:12.663028 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:12.663031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:12.663153 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:12.663597 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:12.663603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:12.663606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:12.663882 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:12.664042 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:12.664820 (+ 778us) server_negotiation.cc:299] Negotiation successful
0810 23:32:12.665052 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":299,"threads_started":1}
I20260810 23:32:12.666376 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:12.646912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:38987 (local address 127.9.48.193:49985)
0810 23:32:12.647964 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:12.648021 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:12.649928 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:12.650637 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:12.650645 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:12.651144 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:12.651702 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:12.651724 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:12.653239 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:12.653243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:12.654445 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:12.654457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:12.655546 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:12.655551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:12.655746 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:12.656460 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:12.656482 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:12.659339 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:12.662434 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:12.662444 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:12.662460 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:12.662885 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:12.663268 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:12.663273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:12.663277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:12.663479 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:12.664016 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:12.664024 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:12.664368 (+ 344us) client_negotiation.cc:769] Sending connection context
0810 23:32:12.664688 (+ 320us) client_negotiation.cc:240] Negotiation successful
0810 23:32:12.665225 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":464,"threads_started":1}
I20260810 23:32:12.668159 14821 heartbeater.cc:347] Connected to a master server at 127.9.48.254:38987
I20260810 23:32:12.668514 14821 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:12.669508 14821 heartbeater.cc:511] Master 127.9.48.254:38987 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
I20260810 23:32:12.671849 14504 ts_manager.cc:194] Registered new tserver with Master: 34a5c14e39574e4484c965c7e54dd711 (127.9.48.193:35313)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
I20260810 23:32:12.673591 14504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:49985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:32:12.700229 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:38987
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:36339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:13.039590 14829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:13.039924 14829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:13.040093 14829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:13.047246 14829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:13.047402 14829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:13.047576 14829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:32:13.057214 14829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:36339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:38987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:13.058548 14829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:13.059917 14829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:13.073760 14841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:13.073840 14847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:13.073798 14843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:13.074992 14829 server_base.cc:1034] running on GCE node
I20260810 23:32:13.075649 14829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:13.076639 14829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:13.077894 14829 hybrid_clock.cc:648] HybridClock initialized: now 1786404733077801 us; error 94 us; skew 500 ppm
I20260810 23:32:13.078266 14829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:13 dist-test-slave-04bb krb5kdc[14420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404733, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:13.081736 14829 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:32:13.083197 14829 webserver.cc:460] Webserver started at http://127.9.48.194:42755/ using document root <none> and password file <none>
I20260810 23:32:13.083842 14829 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:13.083945 14829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:13.084201 14829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:13.087148 14829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "250c7c9ba1a240a89bf1c13b1966e926"
format_stamp: "Formatted at 2026-08-10 23:32:13 on dist-test-slave-04bb"
server_key: "2d0973326f34ce7cd7bc5a0798f16141"
server_key_iv: "df8c4ae3e011dd9e03e08fbaa10b16f5"
server_key_version: "encryptionkey@0"
I20260810 23:32:13.087898 14829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "250c7c9ba1a240a89bf1c13b1966e926"
format_stamp: "Formatted at 2026-08-10 23:32:13 on dist-test-slave-04bb"
server_key: "2d0973326f34ce7cd7bc5a0798f16141"
server_key_iv: "df8c4ae3e011dd9e03e08fbaa10b16f5"
server_key_version: "encryptionkey@0"
I20260810 23:32:13.093163 14829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:32:13.096751 14856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:13.098191 14829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:32:13.098331 14829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "250c7c9ba1a240a89bf1c13b1966e926"
format_stamp: "Formatted at 2026-08-10 23:32:13 on dist-test-slave-04bb"
server_key: "2d0973326f34ce7cd7bc5a0798f16141"
server_key_iv: "df8c4ae3e011dd9e03e08fbaa10b16f5"
server_key_version: "encryptionkey@0"
I20260810 23:32:13.098500 14829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:13.114928 14829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:13.118918 14829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:13.119175 14829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:13.120236 14829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:13.121609 14829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:13.121688 14829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:13.121850 14829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:13.121927 14829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:13.156167 14829 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:37085
I20260810 23:32:13.156212 15041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:37085 every 8 connection(s)
I20260810 23:32:13.157526 14829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:32:13.162797 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 14829
I20260810 23:32:13.162977 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:32:13.163456 9411 external_mini_cluster.cc:1428] Setting key 07235918451ee456fd96702db2db4b6b
Aug 10 23:32:13 dist-test-slave-04bb krb5kdc[14420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404733, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:13.178925 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:13.160159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:47285 (local address 127.9.48.254:38987)
0810 23:32:13.160345 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:13.160349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:13.161642 (+ 1293us) server_negotiation.cc:407] Connection header received
0810 23:32:13.163603 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:13.163609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:13.163721 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:13.163837 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:13.165649 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.166355 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.168135 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.168618 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.168757 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:13.172273 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:13.172278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:13.172282 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:13.172315 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:13.175568 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:13.176514 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:13.176518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:13.176521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:13.176720 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:13.177235 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:13.177238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:13.177240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:13.177493 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:13.177598 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:13.178495 (+ 897us) server_negotiation.cc:299] Negotiation successful
0810 23:32:13.178666 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:32:13.180606 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:13.160721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:38987 (local address 127.9.48.194:47285)
0810 23:32:13.161471 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:13.161510 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:13.163321 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:13.164094 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:13.164102 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:13.164787 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:13.165475 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:13.165496 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.166528 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.166532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:13.167985 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:13.167998 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.168864 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.168871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:13.169100 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:13.169798 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:13.169827 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:13.172042 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:13.175710 (+ 3668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:13.175722 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:13.175742 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:13.176379 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:13.176838 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:13.176846 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:13.176852 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:13.177112 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:13.177635 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:13.177646 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:13.178095 (+ 449us) client_negotiation.cc:769] Sending connection context
0810 23:32:13.178426 (+ 331us) client_negotiation.cc:240] Negotiation successful
0810 23:32:13.179061 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":294,"threads_started":1}
I20260810 23:32:13.182235 15042 heartbeater.cc:347] Connected to a master server at 127.9.48.254:38987
I20260810 23:32:13.182557 15042 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:13.183319 15042 heartbeater.cc:511] Master 127.9.48.254:38987 requested a full tablet report, sending...
I20260810 23:32:13.184939 14504 ts_manager.cc:194] Registered new tserver with Master: 250c7c9ba1a240a89bf1c13b1966e926 (127.9.48.194:37085)
I20260810 23:32:13.185786 14504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:47285
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:32:13.221199 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:38987
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:36339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:13.559957 15049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:13.560213 15049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:13.560400 15049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:13.567473 15049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:13.567567 15049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:13.567740 15049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:32:13.577383 15049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:36339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:38987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:13.578625 15049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:13.579967 15049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:13.593214 15058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:13.593312 15060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:13.593254 15066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:13.594462 15049 server_base.cc:1034] running on GCE node
I20260810 23:32:13.595032 15049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:13.595925 15049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:13.597116 15049 hybrid_clock.cc:648] HybridClock initialized: now 1786404733597076 us; error 38 us; skew 500 ppm
I20260810 23:32:13.597445 15049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:13 dist-test-slave-04bb krb5kdc[14420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404733, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:13.600601 15049 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:32:13.602139 15049 webserver.cc:460] Webserver started at http://127.9.48.195:37293/ using document root <none> and password file <none>
I20260810 23:32:13.602843 15049 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:13.602947 15049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:13.603277 15049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:13.606181 15049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "3e92a36435a84e42a3461d3acd31d8c7"
format_stamp: "Formatted at 2026-08-10 23:32:13 on dist-test-slave-04bb"
server_key: "559b1c84900321bcee18aec6ba24931d"
server_key_iv: "66f307d61aa01fa6e7fac9fb0e4f3a13"
server_key_version: "encryptionkey@0"
I20260810 23:32:13.606920 15049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "3e92a36435a84e42a3461d3acd31d8c7"
format_stamp: "Formatted at 2026-08-10 23:32:13 on dist-test-slave-04bb"
server_key: "559b1c84900321bcee18aec6ba24931d"
server_key_iv: "66f307d61aa01fa6e7fac9fb0e4f3a13"
server_key_version: "encryptionkey@0"
I20260810 23:32:13.611984 15049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:32:13.615614 15079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:13.617018 15049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:32:13.617146 15049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "3e92a36435a84e42a3461d3acd31d8c7"
format_stamp: "Formatted at 2026-08-10 23:32:13 on dist-test-slave-04bb"
server_key: "559b1c84900321bcee18aec6ba24931d"
server_key_iv: "66f307d61aa01fa6e7fac9fb0e4f3a13"
server_key_version: "encryptionkey@0"
I20260810 23:32:13.617300 15049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:13.634965 15049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:13.639359 15049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:13.639690 15049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:13.640641 15049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:13.642026 15049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:13.642100 15049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:13.642266 15049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:13.642341 15049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:13.677270 14821 heartbeater.cc:503] Master 127.9.48.254:38987 was elected leader, sending a full tablet report...
I20260810 23:32:13.677838 15049 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:34583
I20260810 23:32:13.677883 15274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:34583 every 8 connection(s)
I20260810 23:32:13.679436 15049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:32:13.683960 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 15049
I20260810 23:32:13.684175 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:32:13.684732 9411 external_mini_cluster.cc:1428] Setting key 7fb136aeba290b96c43284ec900eb937
Aug 10 23:32:13 dist-test-slave-04bb krb5kdc[14420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404733, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:13.699798 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:13.682208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:46451 (local address 127.9.48.254:38987)
0810 23:32:13.682607 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:13.682611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:13.683749 (+ 1138us) server_negotiation.cc:407] Connection header received
0810 23:32:13.685921 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:13.685927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:13.686020 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:13.686145 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:13.687721 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.688511 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.690071 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.690502 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.690637 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:13.693994 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:13.694007 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:13.694011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:13.694042 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:13.696779 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:13.697492 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:13.697498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:13.697501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:13.697633 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:13.698052 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:13.698058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:13.698062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:13.698332 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:13.698505 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:13.699194 (+ 689us) server_negotiation.cc:299] Negotiation successful
0810 23:32:13.699410 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":197,"threads_started":1}
I20260810 23:32:13.700802 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:13.682824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:38987 (local address 127.9.48.195:46451)
0810 23:32:13.683578 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:13.683635 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:13.685666 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:13.686374 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:13.686382 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:13.686879 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:13.687527 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:13.687554 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.688707 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.688712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:13.689923 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:13.689935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.690738 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.690746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:13.690965 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:13.691568 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:13.691594 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:13.693799 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:13.696926 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:13.696935 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:13.696956 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:13.697358 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:13.697745 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:13.697750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:13.697754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:13.697935 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:13.698486 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:13.698494 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:13.698810 (+ 316us) client_negotiation.cc:769] Sending connection context
0810 23:32:13.699082 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 23:32:13.699582 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":311,"threads_started":1}
I20260810 23:32:13.702078 15275 heartbeater.cc:347] Connected to a master server at 127.9.48.254:38987
I20260810 23:32:13.702450 15275 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:13.703152 15275 heartbeater.cc:511] Master 127.9.48.254:38987 requested a full tablet report, sending...
I20260810 23:32:13.704591 14504 ts_manager.cc:194] Registered new tserver with Master: 3e92a36435a84e42a3461d3acd31d8c7 (127.9.48.195:34583)
I20260810 23:32:13.705407 14504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:46451
I20260810 23:32:13.711969 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:32:13 dist-test-slave-04bb krb5kdc[14420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404733, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 23:32:13 dist-test-slave-04bb krb5kdc[14420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404733, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:13.766857 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:13.753412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.9.48.254:38987)
0810 23:32:13.753649 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:13.753656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:13.753855 (+ 199us) server_negotiation.cc:407] Connection header received
0810 23:32:13.754016 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:13.754020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:13.754100 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:13.754188 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:13.755492 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.756223 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.757528 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.757920 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.758011 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:13.762105 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:13.762111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:13.762114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:13.762147 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:13.764708 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:13.765271 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:13.765275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:13.765277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:13.765404 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:13.765723 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:13.765727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:13.765729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:13.766030 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:13.766212 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:13.766473 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 23:32:13.766633 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260810 23:32:13.767452 14504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47586
I20260810 23:32:13.769342 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 14621
Aug 10 23:32:13 dist-test-slave-04bb krb5kdc[14420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404733, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:32:13.784852 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:13.770395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40512 (local address 127.9.48.193:35313)
0810 23:32:13.771052 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:13.771060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:13.771085 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:32:13.771140 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:13.771146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:13.771384 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:13.771524 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:13.772692 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.773683 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.774815 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.775366 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.775462 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:13.779208 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:13.779220 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:13.779233 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:13.779264 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:13.782493 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:13.783066 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:13.783073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:13.783075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:13.783236 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:13.783561 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:13.783564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:13.783566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:13.783894 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:13.784144 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:13.784307 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 23:32:13.784493 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":347,"threads_started":1}
W20260810 23:32:13.786101 14811 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40512
W20260810 23:32:13.787267 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 14621: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:32:13.787514 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 14621
I20260810 23:32:13.811756 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 14829
Aug 10 23:32:13 dist-test-slave-04bb krb5kdc[14420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404733, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:32:13.827713 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:13.812778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41876 (local address 127.9.48.194:37085)
0810 23:32:13.813306 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:13.813314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:13.813339 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:32:13.813399 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:13.813405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:13.813620 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:13.813830 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:13.815121 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.816140 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.817471 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.818026 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.818172 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:13.822493 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:13.822505 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:13.822531 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:13.822577 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:13.825349 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:13.825958 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:13.825965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:13.825968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:13.826082 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:13.826410 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:13.826413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:13.826415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:13.826731 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:13.826974 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:13.827233 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 23:32:13.827428 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":267,"threads_started":1}
W20260810 23:32:13.829121 15034 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41876
W20260810 23:32:13.830227 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 14829: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:32:13.830394 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 14829
I20260810 23:32:13.869117 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 15049
Aug 10 23:32:13 dist-test-slave-04bb krb5kdc[14420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404733, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:32:13.886642 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:13.870221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58112 (local address 127.9.48.195:34583)
0810 23:32:13.870870 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:13.870878 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:13.870909 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:32:13.870972 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:13.870978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:13.871254 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:13.871389 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:13.872931 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.873991 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.875388 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:13.875980 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:13.876084 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:13.880962 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:13.880975 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:13.880991 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:13.881028 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:13.883795 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:13.884664 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:13.884671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:13.884673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:13.884797 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:13.885146 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:13.885149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:13.885151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:13.885488 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:13.885725 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:13.886006 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 23:32:13.886213 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":322,"threads_started":1}
W20260810 23:32:13.888015 15265 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58112
W20260810 23:32:13.889387 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 15049: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:32:13.889556 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 15049
I20260810 23:32:13.913054 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 14448
I20260810 23:32:13.914340 14561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:14.019383 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 14448
2026-08-10T23:32:14Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4003 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:32:14 dist-test-slave-04bb krb5kdc[15309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:32:14 dist-test-slave-04bb krb5kdc[15309](info): set up 2 sockets
Aug 10 23:32:14 dist-test-slave-04bb krb5kdc[15309](info): commencing operation
krb5kdc: starting...
W20260810 23:32:15.345368 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:32:15 dist-test-slave-04bb krb5kdc[15309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:32:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:32:15.564164 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:36165
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:36165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:15.893255 15336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:15.893538 15336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:15.893709 15336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:15.900928 15336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:15.901026 15336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:15.901125 15336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:15.901226 15336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:15.901304 15336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:15.910974 15336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:36165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:36165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:15.912297 15336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:15.913622 15336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:15.925485 15342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:15.925516 15346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:15.925612 15343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:15.926210 15336 server_base.cc:1034] running on GCE node
I20260810 23:32:15.927011 15336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:15.928418 15336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:15.929673 15336 hybrid_clock.cc:648] HybridClock initialized: now 1786404735929631 us; error 72 us; skew 500 ppm
I20260810 23:32:15.930068 15336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:15 dist-test-slave-04bb krb5kdc[15309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404735, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:15.933214 15336 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:32:15.934657 15336 webserver.cc:460] Webserver started at http://127.9.48.254:35875/ using document root <none> and password file <none>
I20260810 23:32:15.935312 15336 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:15.935411 15336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:15.935683 15336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:15.938606 15336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "0f3e8e42320649ecaedbe455cda84356"
format_stamp: "Formatted at 2026-08-10 23:32:15 on dist-test-slave-04bb"
server_key: "98ed433075278527da38162f85631cd0"
server_key_iv: "900e89594e74f382a62d56d626f3dce8"
server_key_version: "encryptionkey@0"
I20260810 23:32:15.939348 15336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "0f3e8e42320649ecaedbe455cda84356"
format_stamp: "Formatted at 2026-08-10 23:32:15 on dist-test-slave-04bb"
server_key: "98ed433075278527da38162f85631cd0"
server_key_iv: "900e89594e74f382a62d56d626f3dce8"
server_key_version: "encryptionkey@0"
I20260810 23:32:15.944270 15336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:32:15.947750 15358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:15.949249 15336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:32:15.949438 15336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "0f3e8e42320649ecaedbe455cda84356"
format_stamp: "Formatted at 2026-08-10 23:32:15 on dist-test-slave-04bb"
server_key: "98ed433075278527da38162f85631cd0"
server_key_iv: "900e89594e74f382a62d56d626f3dce8"
server_key_version: "encryptionkey@0"
I20260810 23:32:15.949645 15336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:15.983422 15336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:15.987699 15336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:15.988029 15336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:16.007721 15336 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:36165
I20260810 23:32:16.007752 15451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:36165 every 8 connection(s)
I20260810 23:32:16.009258 15336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:16.013074 15452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:16.013806 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 15336
I20260810 23:32:16.014060 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:32:16.014885 9411 external_mini_cluster.cc:1428] Setting key b2c7691a5f0daf0df0123c05af4936fa
I20260810 23:32:16.027621 15452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356: Bootstrap starting.
Aug 10 23:32:16 dist-test-slave-04bb krb5kdc[15309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:16.031373 15452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:16.032435 15452 log.cc:824] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:16.035300 15452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356: No bootstrap required, opened a new log
I20260810 23:32:16.039050 15456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:16.017194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.9.48.254:36165)
0810 23:32:16.018030 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:16.018042 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:16.018089 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:32:16.019771 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:16.019801 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:16.020213 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:16.020648 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:16.022440 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:16.023753 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:16.025634 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:16.026345 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:16.026487 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:16.031263 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:16.031271 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:16.031289 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:16.031332 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:16.034678 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:16.035381 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:16.035388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:16.035415 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:16.035613 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:16.036033 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:16.036038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:16.036041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:16.036661 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:16.037007 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:16.037431 (+ 424us) server_negotiation.cc:299] Negotiation successful
0810 23:32:16.037859 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":390,"threads_started":1}
I20260810 23:32:16.043803 15452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3e8e42320649ecaedbe455cda84356" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 36165 } }
I20260810 23:32:16.044158 15452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:16.044262 15452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3e8e42320649ecaedbe455cda84356, State: Initialized, Role: FOLLOWER
I20260810 23:32:16.044873 15452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3e8e42320649ecaedbe455cda84356" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 36165 } }
I20260810 23:32:16.045163 15452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:32:16.045320 15452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:32:16.045490 15452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:16.047537 15452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3e8e42320649ecaedbe455cda84356" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 36165 } }
I20260810 23:32:16.048084 15452 leader_election.cc:302] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f3e8e42320649ecaedbe455cda84356; no voters:
I20260810 23:32:16.048822 15452 leader_election.cc:288] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:32:16.048995 15461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:16.049724 15461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [term 1 LEADER]: Becoming Leader. State: Replica: 0f3e8e42320649ecaedbe455cda84356, State: Running, Role: LEADER
I20260810 23:32:16.050241 15461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3e8e42320649ecaedbe455cda84356" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 36165 } }
I20260810 23:32:16.050916 15452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:16.054131 15463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f3e8e42320649ecaedbe455cda84356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3e8e42320649ecaedbe455cda84356" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 36165 } } }
I20260810 23:32:16.054596 15463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:16.055163 15465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f3e8e42320649ecaedbe455cda84356. Latest consensus state: current_term: 1 leader_uuid: "0f3e8e42320649ecaedbe455cda84356" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3e8e42320649ecaedbe455cda84356" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 36165 } } }
I20260810 23:32:16.055522 15465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:16.058043 15469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:16.063446 15469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:16.073041 15469 catalog_manager.cc:1347] Generated new cluster ID: bd51d4cd723344dcb0529ddea1b26fef
I20260810 23:32:16.073158 15469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:16.095016 15469 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:32:16.097295 15469 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:16.110081 15469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f3e8e42320649ecaedbe455cda84356: Generated new TSK 0
I20260810 23:32:16.110893 15469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:32:16.180151 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:36165
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:16.543087 15490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:16.543365 15490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:16.543462 15490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:16.550670 15490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:16.550776 15490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:16.550920 15490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:32:16.560842 15490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:36165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:16.562011 15490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:16.563277 15490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:16.576678 15496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:16.576787 15499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:16.576797 15497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:16.578142 15490 server_base.cc:1034] running on GCE node
I20260810 23:32:16.578872 15490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:16.579872 15490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:16.581182 15490 hybrid_clock.cc:648] HybridClock initialized: now 1786404736581129 us; error 38 us; skew 500 ppm
I20260810 23:32:16.581568 15490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:16 dist-test-slave-04bb krb5kdc[15309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404736, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:16.585021 15490 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:32:16.586680 15490 webserver.cc:460] Webserver started at http://127.9.48.193:32953/ using document root <none> and password file <none>
I20260810 23:32:16.587435 15490 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:16.587603 15490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:16.587972 15490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:16.591131 15490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "cc5d268cd3c445259c4ea45e608c2b1b"
format_stamp: "Formatted at 2026-08-10 23:32:16 on dist-test-slave-04bb"
server_key: "e6bb4674cc476e7cb46d8eca47dc67a0"
server_key_iv: "12410aaf55c849e7d673c05498bfb9f8"
server_key_version: "encryptionkey@0"
I20260810 23:32:16.591981 15490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "cc5d268cd3c445259c4ea45e608c2b1b"
format_stamp: "Formatted at 2026-08-10 23:32:16 on dist-test-slave-04bb"
server_key: "e6bb4674cc476e7cb46d8eca47dc67a0"
server_key_iv: "12410aaf55c849e7d673c05498bfb9f8"
server_key_version: "encryptionkey@0"
I20260810 23:32:16.597481 15490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:32:16.601112 15518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:16.602507 15490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:32:16.602710 15490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "cc5d268cd3c445259c4ea45e608c2b1b"
format_stamp: "Formatted at 2026-08-10 23:32:16 on dist-test-slave-04bb"
server_key: "e6bb4674cc476e7cb46d8eca47dc67a0"
server_key_iv: "12410aaf55c849e7d673c05498bfb9f8"
server_key_version: "encryptionkey@0"
I20260810 23:32:16.602941 15490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:16.615577 15490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:16.619728 15490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:16.620033 15490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:16.621078 15490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:16.622447 15490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:16.622566 15490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:16.622726 15490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:16.622809 15490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:16.657851 15490 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:40929
I20260810 23:32:16.657872 15699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:40929 every 8 connection(s)
I20260810 23:32:16.659349 15490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:32:16.665948 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 15490
I20260810 23:32:16.666182 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:32:16.666842 9411 external_mini_cluster.cc:1428] Setting key cc916c5ee66d44569e47a4e06df64d8a
Aug 10 23:32:16 dist-test-slave-04bb krb5kdc[15309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404736, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:16.682039 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:16.661936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:39793 (local address 127.9.48.254:36165)
0810 23:32:16.662387 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:16.662393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:16.663430 (+ 1037us) server_negotiation.cc:407] Connection header received
0810 23:32:16.665350 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:16.665356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:16.665468 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:16.665596 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:16.667369 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:16.668253 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:16.670118 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:16.670885 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:16.671037 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:16.675207 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:16.675212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:16.675215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:16.675254 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:16.678511 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:16.679509 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:16.679513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:16.679516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:16.679659 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:16.680231 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:16.680235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:16.680237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:16.680512 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:16.680685 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:16.681601 (+ 916us) server_negotiation.cc:299] Negotiation successful
0810 23:32:16.681778 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":213,"threads_started":1}
I20260810 23:32:16.683573 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:16.662546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:36165 (local address 127.9.48.193:39793)
0810 23:32:16.663276 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:16.663318 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:16.665068 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:16.665837 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:16.665846 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:16.666537 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:16.667177 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:16.667198 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:16.668457 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:16.668461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:16.669955 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:16.669970 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:16.671145 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:16.671152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:16.671366 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:16.672117 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:16.672142 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:16.674862 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:16.678682 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:16.678689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:16.678702 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:16.679356 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:16.679776 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:16.679785 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:16.679792 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:16.680055 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:16.680696 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:16.680707 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:16.681172 (+ 465us) client_negotiation.cc:769] Sending connection context
0810 23:32:16.681473 (+ 301us) client_negotiation.cc:240] Negotiation successful
0810 23:32:16.682108 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":297,"threads_started":1}
I20260810 23:32:16.685375 15700 heartbeater.cc:347] Connected to a master server at 127.9.48.254:36165
I20260810 23:32:16.685739 15700 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:16.686591 15700 heartbeater.cc:511] Master 127.9.48.254:36165 requested a full tablet report, sending...
I20260810 23:32:16.689077 15389 ts_manager.cc:194] Registered new tserver with Master: cc5d268cd3c445259c4ea45e608c2b1b (127.9.48.193:40929)
I20260810 23:32:16.690681 15389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:39793
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:32:16.722939 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:36165
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:17.060436 15712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:17.060798 15712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:17.060971 15712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:17.068181 15712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:17.068322 15712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:17.068500 15712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:32:17.078083 15712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:36165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:17.079357 15712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:17.080724 15712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:17.093842 15719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:17.094027 15720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:17.094363 15727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:17.095695 15712 server_base.cc:1034] running on GCE node
I20260810 23:32:17.096278 15712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:17.097362 15712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:17.098641 15712 hybrid_clock.cc:648] HybridClock initialized: now 1786404737098553 us; error 52 us; skew 500 ppm
I20260810 23:32:17.099098 15712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:17 dist-test-slave-04bb krb5kdc[15309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404737, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:17.103814 15712 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:32:17.105554 15712 webserver.cc:460] Webserver started at http://127.9.48.194:33919/ using document root <none> and password file <none>
I20260810 23:32:17.106593 15712 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:17.106740 15712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:17.107229 15712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:17.110930 15712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "21072930b2384854995004969e688215"
format_stamp: "Formatted at 2026-08-10 23:32:17 on dist-test-slave-04bb"
server_key: "a111f298a7657bc43efcb4fb9e784f7d"
server_key_iv: "06aa2d0625f10412480b7808f3da0260"
server_key_version: "encryptionkey@0"
I20260810 23:32:17.112023 15712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "21072930b2384854995004969e688215"
format_stamp: "Formatted at 2026-08-10 23:32:17 on dist-test-slave-04bb"
server_key: "a111f298a7657bc43efcb4fb9e784f7d"
server_key_iv: "06aa2d0625f10412480b7808f3da0260"
server_key_version: "encryptionkey@0"
I20260810 23:32:17.118793 15712 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260810 23:32:17.123387 15739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:17.125311 15712 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 23:32:17.125489 15712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "21072930b2384854995004969e688215"
format_stamp: "Formatted at 2026-08-10 23:32:17 on dist-test-slave-04bb"
server_key: "a111f298a7657bc43efcb4fb9e784f7d"
server_key_iv: "06aa2d0625f10412480b7808f3da0260"
server_key_version: "encryptionkey@0"
I20260810 23:32:17.125727 15712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:17.145704 15712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:17.150099 15712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:17.150406 15712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:17.151422 15712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:17.152844 15712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:17.152956 15712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:17.153079 15712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:17.153124 15712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:17.187029 15712 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:34535
I20260810 23:32:17.187120 15951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:34535 every 8 connection(s)
I20260810 23:32:17.188362 15712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:32:17.198066 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 15712
I20260810 23:32:17.198271 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:32:17.198730 9411 external_mini_cluster.cc:1428] Setting key 8b3bd8b28d4f51ee14d69ed1b4526557
Aug 10 23:32:17 dist-test-slave-04bb krb5kdc[15309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404737, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:17.209877 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:17.191243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:44993 (local address 127.9.48.254:36165)
0810 23:32:17.191717 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:17.191723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:17.192759 (+ 1036us) server_negotiation.cc:407] Connection header received
0810 23:32:17.194524 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:17.194528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:17.194597 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:17.194696 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:17.196175 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.196985 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.198642 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.199123 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.199263 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:17.203748 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:17.203753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:17.203757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:17.203789 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:17.206553 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:17.207535 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:17.207539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:17.207541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:17.207666 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:17.208160 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:17.208164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:17.208166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:17.208420 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:17.208608 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:17.209414 (+ 806us) server_negotiation.cc:299] Negotiation successful
0810 23:32:17.209584 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":224,"threads_started":1}
I20260810 23:32:17.211393 15958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:17.191799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:36165 (local address 127.9.48.194:44993)
0810 23:32:17.192577 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:17.192648 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:17.194246 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:17.194905 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:17.194913 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:17.195447 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:17.195993 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:17.196010 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.197131 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.197135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:17.198496 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:17.198509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.199368 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.199375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:17.199668 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:17.200490 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:17.200532 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:17.203511 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:17.206694 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:17.206723 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:17.206740 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:17.207394 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:17.207778 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:17.207785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:17.207791 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:17.208049 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:17.208579 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:17.208589 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:17.209033 (+ 444us) client_negotiation.cc:769] Sending connection context
0810 23:32:17.209352 (+ 319us) client_negotiation.cc:240] Negotiation successful
0810 23:32:17.209923 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":283,"threads_started":1}
I20260810 23:32:17.212978 15953 heartbeater.cc:347] Connected to a master server at 127.9.48.254:36165
I20260810 23:32:17.213305 15953 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:17.214203 15953 heartbeater.cc:511] Master 127.9.48.254:36165 requested a full tablet report, sending...
I20260810 23:32:17.215850 15389 ts_manager.cc:194] Registered new tserver with Master: 21072930b2384854995004969e688215 (127.9.48.194:34535)
I20260810 23:32:17.216787 15389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:44993
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:32:17.254024 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:36165
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:17.602480 15961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:17.602778 15961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:17.602941 15961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:17.610133 15961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:17.610280 15961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:17.610446 15961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:32:17.620010 15961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:36165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:17.621455 15961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:17.622846 15961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:17.636502 15969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:17.636602 15970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:17.636634 15974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:17.637897 15961 server_base.cc:1034] running on GCE node
I20260810 23:32:17.638546 15961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:17.639518 15961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:17.640745 15961 hybrid_clock.cc:648] HybridClock initialized: now 1786404737640705 us; error 54 us; skew 500 ppm
I20260810 23:32:17.641070 15961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:17 dist-test-slave-04bb krb5kdc[15309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404737, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:17.644263 15961 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:32:17.645836 15961 webserver.cc:460] Webserver started at http://127.9.48.195:34435/ using document root <none> and password file <none>
I20260810 23:32:17.646538 15961 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:17.646690 15961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:17.647133 15961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:17.650139 15961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "abb9ebc9e9a346dea994d442c596280e"
format_stamp: "Formatted at 2026-08-10 23:32:17 on dist-test-slave-04bb"
server_key: "5f73a864d0ee2c679d2ecf00db89ec26"
server_key_iv: "1d493baebaa1f78262bebe845433915f"
server_key_version: "encryptionkey@0"
I20260810 23:32:17.650936 15961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "abb9ebc9e9a346dea994d442c596280e"
format_stamp: "Formatted at 2026-08-10 23:32:17 on dist-test-slave-04bb"
server_key: "5f73a864d0ee2c679d2ecf00db89ec26"
server_key_iv: "1d493baebaa1f78262bebe845433915f"
server_key_version: "encryptionkey@0"
I20260810 23:32:17.656072 15961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:32:17.660372 15988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:17.661970 15961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:32:17.662130 15961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "abb9ebc9e9a346dea994d442c596280e"
format_stamp: "Formatted at 2026-08-10 23:32:17 on dist-test-slave-04bb"
server_key: "5f73a864d0ee2c679d2ecf00db89ec26"
server_key_iv: "1d493baebaa1f78262bebe845433915f"
server_key_version: "encryptionkey@0"
I20260810 23:32:17.662341 15961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:17.680775 15961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:17.684783 15961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:17.685017 15961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:17.685889 15961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:17.687211 15961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:17.687284 15961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:17.687379 15961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:17.687426 15961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:17.693885 15700 heartbeater.cc:503] Master 127.9.48.254:36165 was elected leader, sending a full tablet report...
I20260810 23:32:17.722625 15961 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:43307
I20260810 23:32:17.722730 16172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:43307 every 8 connection(s)
I20260810 23:32:17.724438 15961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:32:17.726382 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 15961
I20260810 23:32:17.726599 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:32:17.727135 9411 external_mini_cluster.cc:1428] Setting key 7559824efac4064db704e52af1a3c60c
Aug 10 23:32:17 dist-test-slave-04bb krb5kdc[15309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404737, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:17.745540 16176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:17.727571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:53445 (local address 127.9.48.254:36165)
0810 23:32:17.728098 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:17.728105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:17.729126 (+ 1021us) server_negotiation.cc:407] Connection header received
0810 23:32:17.731335 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:17.731339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:17.731409 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:17.731497 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:17.733006 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.733774 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.735319 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.735697 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.735786 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:17.739277 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:17.739282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:17.739285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:17.739316 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:17.742319 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:17.743287 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:17.743291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:17.743293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:17.743411 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:17.743799 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:17.743810 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:17.743812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:17.744064 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:17.744183 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:17.745100 (+ 917us) server_negotiation.cc:299] Negotiation successful
0810 23:32:17.745263 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":669,"threads_started":1}
I20260810 23:32:17.746506 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:17.728025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:36165 (local address 127.9.48.195:53445)
0810 23:32:17.728948 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:17.729001 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:17.731036 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:17.731684 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:17.731692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:17.732210 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:17.732860 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:17.732877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.733938 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.733942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:17.735194 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:17.735206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.735885 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.735890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:17.736075 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:17.736844 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:17.736872 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:17.739073 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:17.742483 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:17.742493 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:17.742511 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:17.743152 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:17.743506 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:17.743509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:17.743512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:17.743698 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:17.744237 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:17.744281 (+ 44us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:17.744735 (+ 454us) client_negotiation.cc:769] Sending connection context
0810 23:32:17.744993 (+ 258us) client_negotiation.cc:240] Negotiation successful
0810 23:32:17.745420 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":355,"threads_started":1}
I20260810 23:32:17.747921 16173 heartbeater.cc:347] Connected to a master server at 127.9.48.254:36165
I20260810 23:32:17.748232 16173 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:17.749099 16173 heartbeater.cc:511] Master 127.9.48.254:36165 requested a full tablet report, sending...
I20260810 23:32:17.750607 15389 ts_manager.cc:194] Registered new tserver with Master: abb9ebc9e9a346dea994d442c596280e (127.9.48.195:43307)
I20260810 23:32:17.751503 15389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:53445
I20260810 23:32:17.755209 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:32:17 dist-test-slave-04bb krb5kdc[15309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:32:17 dist-test-slave-04bb krb5kdc[15309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:32:17.813287 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:17.797260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.9.48.193:40929)
0810 23:32:17.797967 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:17.797977 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:17.798011 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:32:17.798096 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:17.798102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:17.798324 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:17.798529 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:17.799980 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.801120 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.802365 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.803183 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.803329 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:17.807637 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:17.807649 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:17.807664 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:17.807703 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:17.810821 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:17.811458 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:17.811464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:17.811467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:17.811584 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:17.811918 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:17.811921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:17.811923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:17.812249 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:17.812493 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:17.812789 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 23:32:17.812997 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":324,"threads_started":1}
Aug 10 23:32:17 dist-test-slave-04bb krb5kdc[15309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:17.831480 16176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:17.817866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.9.48.254:36165)
0810 23:32:17.818085 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:17.818091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:17.818198 (+ 107us) server_negotiation.cc:407] Connection header received
0810 23:32:17.818396 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:17.818406 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:17.818474 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:17.818579 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:17.819844 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.820636 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.821876 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.822286 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.822436 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:17.826234 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:17.826240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:17.826243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:17.826276 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:17.828872 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:17.829569 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:17.829573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:17.829575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:17.829693 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:17.830015 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:17.830020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:17.830024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:17.830582 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:17.830804 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:17.831062 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 23:32:17.831256 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 23:32:17.832103 15389 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60352
I20260810 23:32:17.833192 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 15490
I20260810 23:32:17.845280 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:17.834148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34166 (local address 127.9.48.193:40929)
0810 23:32:17.834313 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:17.834317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:17.834551 (+ 234us) server_negotiation.cc:407] Connection header received
0810 23:32:17.834667 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:17.834671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:17.834733 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:17.834809 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:17.835964 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.836696 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.837835 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.838257 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.838354 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:17.840509 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:17.840514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:17.840517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:17.840614 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:17.843093 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:17.843680 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:17.843684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:17.843686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:17.843810 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:17.844122 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:17.844126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:17.844128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:17.844377 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:17.844641 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:17.844881 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 23:32:17.845051 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:32:17.845937 15692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:17.928514 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 15490
I20260810 23:32:17.955199 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 15712
Aug 10 23:32:17 dist-test-slave-04bb krb5kdc[15309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:32:17.971709 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:17.956221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.9.48.194:34535)
0810 23:32:17.956871 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:17.956879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:17.956907 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:32:17.956974 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:17.956981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:17.957253 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:17.957410 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:17.958715 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.959893 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.961319 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:17.961882 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:17.961991 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:17.966150 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:17.966161 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:17.966175 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:17.966213 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:17.969333 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:17.969979 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:17.969986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:17.969989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:17.970112 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:17.970429 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:17.970432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:17.970434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:17.970758 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:17.970993 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:17.971217 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 23:32:17.971417 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":359,"threads_started":1}
I20260810 23:32:17.973285 15946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:18.057245 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 15712
I20260810 23:32:18.082557 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 15961
Aug 10 23:32:18 dist-test-slave-04bb krb5kdc[15309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:32:18.101289 16195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:18.083647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58210 (local address 127.9.48.195:43307)
0810 23:32:18.084238 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:18.084245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:18.084274 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:32:18.084338 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:18.084344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:18.084648 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:18.084781 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:18.086023 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:18.087053 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:18.088317 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:18.089060 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:18.089164 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:18.094621 (+ 5457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:18.094640 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:18.094661 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:18.094713 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:18.098484 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:18.099313 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:18.099321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:18.099323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:18.099447 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:18.099779 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:18.099782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:18.099784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:18.100119 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:18.100355 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:18.100668 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 23:32:18.100881 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":270,"threads_started":1}
I20260810 23:32:18.102562 16162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:18.186077 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 15961
I20260810 23:32:18.211514 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 15336
I20260810 23:32:18.212786 15444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:18.319038 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 15336
2026-08-10T23:32:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4295 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:32:18 dist-test-slave-04bb krb5kdc[16202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:32:18 dist-test-slave-04bb krb5kdc[16202](info): set up 2 sockets
Aug 10 23:32:18 dist-test-slave-04bb krb5kdc[16202](info): commencing operation
krb5kdc: starting...
W20260810 23:32:19.648118 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.277s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:32:19 dist-test-slave-04bb krb5kdc[16202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:32:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:32:19.885145 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:40813
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:40813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:20.222160 16220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:20.222429 16220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:20.222554 16220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:20.229671 16220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:20.229769 16220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:20.229872 16220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:20.229952 16220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:20.230028 16220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:20.239524 16220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:40813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:40813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:20.240837 16220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:20.242165 16220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:20.253631 16233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:20.253712 16228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:20.253684 16231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:20.254528 16220 server_base.cc:1034] running on GCE node
I20260810 23:32:20.255303 16220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:20.256673 16220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:20.257917 16220 hybrid_clock.cc:648] HybridClock initialized: now 1786404740257867 us; error 54 us; skew 500 ppm
I20260810 23:32:20.258303 16220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:20 dist-test-slave-04bb krb5kdc[16202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404740, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:20.261520 16220 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:32:20.262735 16220 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:20.262872 16220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:20.263231 16220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:20.266328 16220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "aeaa2874892446d1806026f3f15b4068"
format_stamp: "Formatted at 2026-08-10 23:32:20 on dist-test-slave-04bb"
server_key: "fb85fda0c34c8eff99f1db5246fd95bf"
server_key_iv: "5c5d0535c5b78c386883e06a38086092"
server_key_version: "encryptionkey@0"
I20260810 23:32:20.267153 16220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "aeaa2874892446d1806026f3f15b4068"
format_stamp: "Formatted at 2026-08-10 23:32:20 on dist-test-slave-04bb"
server_key: "fb85fda0c34c8eff99f1db5246fd95bf"
server_key_iv: "5c5d0535c5b78c386883e06a38086092"
server_key_version: "encryptionkey@0"
I20260810 23:32:20.272118 16220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:32:20.275645 16240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:20.277071 16220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:20.277297 16220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "aeaa2874892446d1806026f3f15b4068"
format_stamp: "Formatted at 2026-08-10 23:32:20 on dist-test-slave-04bb"
server_key: "fb85fda0c34c8eff99f1db5246fd95bf"
server_key_iv: "5c5d0535c5b78c386883e06a38086092"
server_key_version: "encryptionkey@0"
I20260810 23:32:20.277525 16220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:20.294696 16220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:20.298679 16220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:20.298957 16220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:20.317826 16220 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:40813
I20260810 23:32:20.317852 16328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:40813 every 8 connection(s)
I20260810 23:32:20.318817 16220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:20.322480 16329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:20.327919 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 16220
I20260810 23:32:20.328158 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:32:20.329010 9411 external_mini_cluster.cc:1428] Setting key d1afd78ae966a4d5b3dbf1786cd7bf95
I20260810 23:32:20.334105 16329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068: Bootstrap starting.
I20260810 23:32:20.337811 16329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:20.339125 16329 log.cc:824] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:20.342540 16329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068: No bootstrap required, opened a new log
Aug 10 23:32:20 dist-test-slave-04bb krb5kdc[16202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:20.351029 16329 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeaa2874892446d1806026f3f15b4068" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40813 } }
I20260810 23:32:20.351423 16329 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:20.351552 16329 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeaa2874892446d1806026f3f15b4068, State: Initialized, Role: FOLLOWER
I20260810 23:32:20.352159 16329 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeaa2874892446d1806026f3f15b4068" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40813 } }
I20260810 23:32:20.352421 16329 raft_consensus.cc:388] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:32:20.352630 16329 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:32:20.352834 16329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:20.354372 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:20.331237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41686 (local address 127.9.48.254:40813)
0810 23:32:20.332108 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:20.332120 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:20.332165 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:32:20.333911 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:20.333935 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:20.334370 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:20.334676 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:20.336674 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:20.338003 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:20.339882 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:20.340928 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:20.341058 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:20.345921 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:20.345929 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:20.345949 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:20.345994 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:20.350015 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:20.350688 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:20.350703 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:20.350709 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:20.350917 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:20.351309 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:20.351317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:20.351321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:20.352141 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:20.352446 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:20.352927 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 23:32:20.353358 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":401,"threads_started":1}
I20260810 23:32:20.355557 16329 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeaa2874892446d1806026f3f15b4068" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40813 } }
I20260810 23:32:20.356381 16329 leader_election.cc:302] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aeaa2874892446d1806026f3f15b4068; no voters:
I20260810 23:32:20.357121 16329 leader_election.cc:288] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:32:20.357266 16336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:20.358076 16336 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [term 1 LEADER]: Becoming Leader. State: Replica: aeaa2874892446d1806026f3f15b4068, State: Running, Role: LEADER
I20260810 23:32:20.358625 16336 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeaa2874892446d1806026f3f15b4068" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40813 } }
I20260810 23:32:20.359504 16329 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:20.361776 16340 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeaa2874892446d1806026f3f15b4068. Latest consensus state: current_term: 1 leader_uuid: "aeaa2874892446d1806026f3f15b4068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeaa2874892446d1806026f3f15b4068" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40813 } } }
I20260810 23:32:20.363428 16339 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aeaa2874892446d1806026f3f15b4068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeaa2874892446d1806026f3f15b4068" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40813 } } }
I20260810 23:32:20.364176 16339 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:20.364185 16340 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:20.368168 16355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:20.373800 16355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:20.382555 16355 catalog_manager.cc:1347] Generated new cluster ID: 71d9c2aa8bc8468c9e2277a0d9abc9db
I20260810 23:32:20.382678 16355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:20.405939 16355 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:32:20.407289 16355 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:20.414214 16355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068: Generated new TSK 0
I20260810 23:32:20.414999 16355 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:32:20.474355 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40813
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:20.812479 16377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:20.812824 16377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:20.813020 16377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:20.820077 16377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:20.820178 16377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:20.820361 16377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:32:20.830103 16377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:20.831429 16377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:20.832875 16377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:20.846293 16385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:20.846330 16387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:20.846364 16384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:20.847122 16377 server_base.cc:1034] running on GCE node
I20260810 23:32:20.847790 16377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:20.848796 16377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:20.850032 16377 hybrid_clock.cc:648] HybridClock initialized: now 1786404740849970 us; error 56 us; skew 500 ppm
I20260810 23:32:20.850358 16377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:20 dist-test-slave-04bb krb5kdc[16202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404740, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:20.853639 16377 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:32:20.854866 16377 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:20.855002 16377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:20.855312 16377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:20.858371 16377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "f385bc249b3d45e2ab894435ee55de85"
format_stamp: "Formatted at 2026-08-10 23:32:20 on dist-test-slave-04bb"
server_key: "290a1a00375cd5e40e3b5fdce8dbf90b"
server_key_iv: "422825b0f80bd0222f44df9458bece05"
server_key_version: "encryptionkey@0"
I20260810 23:32:20.859189 16377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "f385bc249b3d45e2ab894435ee55de85"
format_stamp: "Formatted at 2026-08-10 23:32:20 on dist-test-slave-04bb"
server_key: "290a1a00375cd5e40e3b5fdce8dbf90b"
server_key_iv: "422825b0f80bd0222f44df9458bece05"
server_key_version: "encryptionkey@0"
I20260810 23:32:20.864398 16377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:32:20.867947 16397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:20.869524 16377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:20.869704 16377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "f385bc249b3d45e2ab894435ee55de85"
format_stamp: "Formatted at 2026-08-10 23:32:20 on dist-test-slave-04bb"
server_key: "290a1a00375cd5e40e3b5fdce8dbf90b"
server_key_iv: "422825b0f80bd0222f44df9458bece05"
server_key_version: "encryptionkey@0"
I20260810 23:32:20.869941 16377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:20.884011 16377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:20.888065 16377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:20.888345 16377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:20.889348 16377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:20.890758 16377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:20.890887 16377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:20.891023 16377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:20.891112 16377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:20.924770 16377 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:46821
I20260810 23:32:20.924823 16588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:46821 every 8 connection(s)
I20260810 23:32:20.925719 16377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:32:20.935914 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 16377
I20260810 23:32:20.936201 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:32:20.937152 9411 external_mini_cluster.cc:1428] Setting key 0320302a1d76ffce241175f6c2f1d321
Aug 10 23:32:20 dist-test-slave-04bb krb5kdc[16202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404740, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:20.945899 16593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:20.928255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:33429 (local address 127.9.48.254:40813)
0810 23:32:20.928732 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:20.928738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:20.929688 (+ 950us) server_negotiation.cc:407] Connection header received
0810 23:32:20.931537 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:20.931542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:20.931639 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:20.931761 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:20.933320 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:20.934048 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:20.935616 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:20.936019 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:20.936165 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:20.940220 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:20.940227 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:20.940231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:20.940276 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:20.942848 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:20.943583 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:20.943589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:20.943593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:20.943718 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:20.944146 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:20.944151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:20.944154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:20.944424 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:20.944603 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:20.945295 (+ 692us) server_negotiation.cc:299] Negotiation successful
0810 23:32:20.945518 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":270,"threads_started":1}
I20260810 23:32:20.946785 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:20.928816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40813 (local address 127.9.48.193:33429)
0810 23:32:20.929522 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:20.929568 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:20.931267 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:20.931997 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:20.932005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:20.932508 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:20.933137 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:20.933155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:20.934215 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:20.934225 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:20.935460 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:20.935473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:20.936335 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:20.936347 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:20.936695 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:20.937496 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:20.937534 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:20.940011 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:20.942994 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:20.943002 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:20.943017 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:20.943451 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:20.943836 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:20.943841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:20.943845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:20.944021 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:20.944581 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:20.944589 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:20.944919 (+ 330us) client_negotiation.cc:769] Sending connection context
0810 23:32:20.945182 (+ 263us) client_negotiation.cc:240] Negotiation successful
0810 23:32:20.945690 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":286,"threads_started":1}
I20260810 23:32:20.948424 16590 heartbeater.cc:347] Connected to a master server at 127.9.48.254:40813
I20260810 23:32:20.948757 16590 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:20.949574 16590 heartbeater.cc:511] Master 127.9.48.254:40813 requested a full tablet report, sending...
I20260810 23:32:20.951717 16272 ts_manager.cc:194] Registered new tserver with Master: f385bc249b3d45e2ab894435ee55de85 (127.9.48.193:46821)
I20260810 23:32:20.953222 16272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:33429
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:32:20.992175 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40813
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:21.329519 16599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:21.329823 16599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:21.330006 16599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:21.337271 16599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:21.337414 16599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:21.337597 16599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:32:21.347344 16599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:21.348780 16599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:21.350145 16599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:21.364724 16608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:21.365034 16611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:21.365510 16605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:21.368752 16599 server_base.cc:1034] running on GCE node
I20260810 23:32:21.369361 16599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:21.370882 16599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:21.372131 16599 hybrid_clock.cc:648] HybridClock initialized: now 1786404741372060 us; error 51 us; skew 500 ppm
I20260810 23:32:21.372586 16599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:21 dist-test-slave-04bb krb5kdc[16202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404741, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:21.376329 16599 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:32:21.377485 16599 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:21.377576 16599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:21.377856 16599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:21.381361 16599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "c151ea3295ff4ad69955ac80e08ada4b"
format_stamp: "Formatted at 2026-08-10 23:32:21 on dist-test-slave-04bb"
server_key: "77f9e199cb41a280f122c614c76b67ab"
server_key_iv: "334381d80f2b6a026e689426343e35ab"
server_key_version: "encryptionkey@0"
I20260810 23:32:21.382251 16599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "c151ea3295ff4ad69955ac80e08ada4b"
format_stamp: "Formatted at 2026-08-10 23:32:21 on dist-test-slave-04bb"
server_key: "77f9e199cb41a280f122c614c76b67ab"
server_key_iv: "334381d80f2b6a026e689426343e35ab"
server_key_version: "encryptionkey@0"
I20260810 23:32:21.387849 16599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 23:32:21.391463 16617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:21.392858 16599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:21.393004 16599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "c151ea3295ff4ad69955ac80e08ada4b"
format_stamp: "Formatted at 2026-08-10 23:32:21 on dist-test-slave-04bb"
server_key: "77f9e199cb41a280f122c614c76b67ab"
server_key_iv: "334381d80f2b6a026e689426343e35ab"
server_key_version: "encryptionkey@0"
I20260810 23:32:21.393224 16599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:21.411031 16599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:21.415059 16599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:21.415280 16599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:21.416133 16599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:21.417485 16599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:21.417589 16599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:21.417666 16599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:21.417725 16599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:21.451452 16599 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:42397
I20260810 23:32:21.451540 16802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:42397 every 8 connection(s)
I20260810 23:32:21.452322 16599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:32:21.455117 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 16599
I20260810 23:32:21.455353 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:32:21.455926 9411 external_mini_cluster.cc:1428] Setting key 5dd3cbb3e16b88aadb08ec3eed414d81
Aug 10 23:32:21 dist-test-slave-04bb krb5kdc[16202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404741, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:21.473443 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:21.454914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:50635 (local address 127.9.48.254:40813)
0810 23:32:21.455388 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:21.455392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:21.456669 (+ 1277us) server_negotiation.cc:407] Connection header received
0810 23:32:21.458808 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:21.458813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:21.458918 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:21.459038 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:21.460771 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:21.461506 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:21.463054 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:21.463501 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:21.463641 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:21.467051 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:21.467057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:21.467060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:21.467093 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:21.469905 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:21.470917 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:21.470921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:21.470924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:21.471114 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:21.471579 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:21.471582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:21.471584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:21.471871 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:21.471983 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:21.472948 (+ 965us) server_negotiation.cc:299] Negotiation successful
0810 23:32:21.473123 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":234,"threads_started":1}
I20260810 23:32:21.475054 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:21.455625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40813 (local address 127.9.48.194:50635)
0810 23:32:21.456450 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:21.456497 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:21.458543 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:21.459315 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:21.459331 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:21.459853 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:21.460592 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:21.460620 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:21.461664 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:21.461670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:21.462905 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:21.462917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:21.463730 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:21.463737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:21.463955 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:21.464687 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:21.464710 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:21.466822 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:21.470065 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:21.470078 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:21.470097 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:21.470781 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:21.471211 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:21.471219 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:21.471225 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:21.471463 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:21.472003 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:21.472013 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:21.472511 (+ 498us) client_negotiation.cc:769] Sending connection context
0810 23:32:21.472856 (+ 345us) client_negotiation.cc:240] Negotiation successful
0810 23:32:21.473448 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":377,"threads_started":1}
I20260810 23:32:21.476577 16804 heartbeater.cc:347] Connected to a master server at 127.9.48.254:40813
I20260810 23:32:21.476908 16804 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:21.477655 16804 heartbeater.cc:511] Master 127.9.48.254:40813 requested a full tablet report, sending...
I20260810 23:32:21.479179 16272 ts_manager.cc:194] Registered new tserver with Master: c151ea3295ff4ad69955ac80e08ada4b (127.9.48.194:42397)
I20260810 23:32:21.479988 16272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:50635
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:32:21.511708 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40813
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:21.840515 16811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:21.840845 16811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:21.840945 16811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:21.847967 16811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:21.848073 16811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:21.848182 16811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:32:21.857810 16811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:21.859228 16811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:21.860668 16811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:21.874379 16819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:21.874348 16824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:21.874835 16827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:21.875554 16811 server_base.cc:1034] running on GCE node
I20260810 23:32:21.876231 16811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:21.877355 16811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:21.878623 16811 hybrid_clock.cc:648] HybridClock initialized: now 1786404741878548 us; error 49 us; skew 500 ppm
I20260810 23:32:21.878988 16811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:21 dist-test-slave-04bb krb5kdc[16202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404741, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:21.882666 16811 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:32:21.883833 16811 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:21.883975 16811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:21.884320 16811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:21.887375 16811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "44dda970f9ff422591c49a6a679b280a"
format_stamp: "Formatted at 2026-08-10 23:32:21 on dist-test-slave-04bb"
server_key: "321e84780304c2798ec63d831c317429"
server_key_iv: "38718ee1cb7d090a1cd301cbc4cc29f7"
server_key_version: "encryptionkey@0"
I20260810 23:32:21.888264 16811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "44dda970f9ff422591c49a6a679b280a"
format_stamp: "Formatted at 2026-08-10 23:32:21 on dist-test-slave-04bb"
server_key: "321e84780304c2798ec63d831c317429"
server_key_iv: "38718ee1cb7d090a1cd301cbc4cc29f7"
server_key_version: "encryptionkey@0"
I20260810 23:32:21.893893 16811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:32:21.897468 16840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:21.898977 16811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 23:32:21.899179 16811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "44dda970f9ff422591c49a6a679b280a"
format_stamp: "Formatted at 2026-08-10 23:32:21 on dist-test-slave-04bb"
server_key: "321e84780304c2798ec63d831c317429"
server_key_iv: "38718ee1cb7d090a1cd301cbc4cc29f7"
server_key_version: "encryptionkey@0"
I20260810 23:32:21.899501 16811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:21.911674 16811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:21.915639 16811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:21.915930 16811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:21.917006 16811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:21.918401 16811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:21.918540 16811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:21.918699 16811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:21.918800 16811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:21.955473 16811 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:39103
I20260810 23:32:21.955577 17013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:39103 every 8 connection(s)
I20260810 23:32:21.956408 16811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:32:21.956924 16590 heartbeater.cc:503] Master 127.9.48.254:40813 was elected leader, sending a full tablet report...
I20260810 23:32:21.961946 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 16811
I20260810 23:32:21.962172 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:32:21.962726 9411 external_mini_cluster.cc:1428] Setting key 1834ae52292ee853a4ec17a9361b5e03
Aug 10 23:32:21 dist-test-slave-04bb krb5kdc[16202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404741, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:21.976740 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:21.959086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:47059 (local address 127.9.48.254:40813)
0810 23:32:21.959329 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:21.959335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:21.960636 (+ 1301us) server_negotiation.cc:407] Connection header received
0810 23:32:21.962532 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:21.962539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:21.962625 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:21.962736 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:21.964567 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:21.965497 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:21.967060 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:21.967459 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:21.967555 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:21.970932 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:21.970940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:21.970947 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:21.970987 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:21.973899 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:21.974570 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:21.974574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:21.974576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:21.974693 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:21.975064 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:21.975067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:21.975069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:21.975363 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:21.975492 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:21.976135 (+ 643us) server_negotiation.cc:299] Negotiation successful
0810 23:32:21.976366 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260810 23:32:21.977890 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:21.959666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40813 (local address 127.9.48.195:47059)
0810 23:32:21.960446 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:21.960493 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:21.962237 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:21.962998 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:21.963005 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:21.963677 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:21.964332 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:21.964352 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:21.965655 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:21.965660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:21.966925 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:21.966938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:21.967648 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:21.967652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:21.967853 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:21.968511 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:21.968582 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:21.970731 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:21.974017 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:21.974024 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:21.974034 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:21.974468 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:21.974788 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:21.974791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:21.974793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:21.974970 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:21.975478 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:21.975483 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:21.975768 (+ 285us) client_negotiation.cc:769] Sending connection context
0810 23:32:21.976003 (+ 235us) client_negotiation.cc:240] Negotiation successful
0810 23:32:21.976452 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":299,"threads_started":1}
I20260810 23:32:21.979187 17016 heartbeater.cc:347] Connected to a master server at 127.9.48.254:40813
I20260810 23:32:21.979451 17016 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:21.980022 17016 heartbeater.cc:511] Master 127.9.48.254:40813 requested a full tablet report, sending...
I20260810 23:32:21.981386 16272 ts_manager.cc:194] Registered new tserver with Master: 44dda970f9ff422591c49a6a679b280a (127.9.48.195:39103)
I20260810 23:32:21.982220 16272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:47059
I20260810 23:32:21.990473 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:32:22.007087 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:21.994902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.9.48.254:40813)
0810 23:32:21.995056 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:21.995061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:21.995621 (+ 560us) server_negotiation.cc:407] Connection header received
0810 23:32:21.995762 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:21.995765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:21.995823 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:21.995900 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:21.997426 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:21.998257 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:21.999415 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:21.999804 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:21.999895 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:22.002168 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:22.002173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:22.002176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:22.002205 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:22.004935 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:22.005511 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:22.005515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:22.005517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:22.005636 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:22.005943 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:22.005947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:22.005949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:22.006239 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:22.006434 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:22.006693 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 23:32:22.006843 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:32:22.010917 16272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:22.017679 16272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:22.050287 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:22.029670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:39103 (local address 127.0.0.1:60424)
0810 23:32:22.041020 (+ 11350us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:22.041074 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:22.041270 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:22.042413 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:22.042421 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:22.042469 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:22.043504 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:22.043516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.045594 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.045600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:22.047654 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:22.047667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.049352 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.049359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:22.049547 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:22.049582 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:32:22.049670 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:32:22.049795 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11084,"mutex_wait_us":540,"spinlock_wait_cycles":5248,"thread_start_us":11079,"threads_started":1}
I20260810 23:32:22.050590 17048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:22.030556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.9.48.194:42397)
0810 23:32:22.040101 (+ 9545us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:22.040113 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:22.041472 (+ 1359us) server_negotiation.cc:407] Connection header received
0810 23:32:22.041595 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:22.041600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:22.041782 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:22.041987 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:22.043514 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.044981 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.047296 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.048801 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.048964 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:22.049180 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:22.049524 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 23:32:22.049701 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9367,"thread_start_us":9213,"threads_started":1}
I20260810 23:32:22.051357 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:22.029756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.9.48.195:39103)
0810 23:32:22.040583 (+ 10827us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:22.040593 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:22.041461 (+ 868us) server_negotiation.cc:407] Connection header received
0810 23:32:22.041578 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:22.041587 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:22.041827 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:22.042019 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:22.043761 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.045448 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.047890 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.049011 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.049138 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:22.049370 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:22.050682 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0810 23:32:22.050956 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10594,"thread_start_us":10392,"threads_started":1}
I20260810 23:32:22.050343 17044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:22.030063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:42397 (local address 127.0.0.1:53258)
0810 23:32:22.041019 (+ 10956us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:22.041071 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:22.041269 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:22.042278 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:22.042286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:22.042316 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:22.043315 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:22.043332 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.045110 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.045115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:22.047075 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:22.047089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.049033 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.049040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:22.049247 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:22.049301 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 23:32:22.049472 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 23:32:22.049665 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10838,"mutex_wait_us":420,"spinlock_wait_cycles":25344,"thread_start_us":10514,"threads_started":1}
I20260810 23:32:22.052340 17051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:22.042392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46821 (local address 127.0.0.1:44764)
0810 23:32:22.043447 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:22.043465 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:22.043574 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:22.044463 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:22.044468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:22.044491 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:22.044855 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:22.044868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.046595 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.046604 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:22.048867 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:22.048884 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.051544 (+ 2660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.051550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:22.051735 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:22.051769 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:32:22.051871 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:32:22.051990 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":1,"thread_start_us":722,"threads_started":1}
I20260810 23:32:22.052556 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:22.041774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44764 (local address 127.9.48.193:46821)
0810 23:32:22.043138 (+ 1364us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:22.043146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:22.043690 (+ 544us) server_negotiation.cc:407] Connection header received
0810 23:32:22.043789 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:22.043798 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:22.044018 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:22.044203 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:22.045039 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.046439 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.049056 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.050922 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.051159 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:22.051389 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:22.051944 (+ 555us) server_negotiation.cc:299] Negotiation successful
0810 23:32:22.052128 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1143,"thread_start_us":954,"threads_started":1}
I20260810 23:32:22.055478 16688 tablet_service.cc:1467] Processing CreateTablet for tablet bad9a5ea46a44dfd8ee6df69d8fc569d (DEFAULT_TABLE table=test-table [id=e360a2e99b8b4a02a2f66a812ac871d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:22.055670 16918 tablet_service.cc:1467] Processing CreateTablet for tablet bad9a5ea46a44dfd8ee6df69d8fc569d (DEFAULT_TABLE table=test-table [id=e360a2e99b8b4a02a2f66a812ac871d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:22.055996 16481 tablet_service.cc:1467] Processing CreateTablet for tablet bad9a5ea46a44dfd8ee6df69d8fc569d (DEFAULT_TABLE table=test-table [id=e360a2e99b8b4a02a2f66a812ac871d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:22.056937 16688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bad9a5ea46a44dfd8ee6df69d8fc569d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:22.057139 16918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bad9a5ea46a44dfd8ee6df69d8fc569d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:22.057353 16481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bad9a5ea46a44dfd8ee6df69d8fc569d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:22.068934 17052 tablet_bootstrap.cc:492] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b: Bootstrap starting.
I20260810 23:32:22.072803 17055 tablet_bootstrap.cc:492] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a: Bootstrap starting.
I20260810 23:32:22.073603 17057 tablet_bootstrap.cc:492] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85: Bootstrap starting.
I20260810 23:32:22.073675 17052 tablet_bootstrap.cc:654] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:22.075004 17052 log.cc:824] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:22.076218 17055 tablet_bootstrap.cc:654] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:22.076682 17057 tablet_bootstrap.cc:654] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:22.077492 17055 log.cc:824] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:22.077626 17057 log.cc:824] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:22.078414 17052 tablet_bootstrap.cc:492] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b: No bootstrap required, opened a new log
I20260810 23:32:22.079341 17052 ts_tablet_manager.cc:1397] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260810 23:32:22.081004 17057 tablet_bootstrap.cc:492] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85: No bootstrap required, opened a new log
I20260810 23:32:22.081281 17057 ts_tablet_manager.cc:1397] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260810 23:32:22.082616 17055 tablet_bootstrap.cc:492] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a: No bootstrap required, opened a new log
I20260810 23:32:22.082863 17055 ts_tablet_manager.cc:1397] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 23:32:22.088485 17057 raft_consensus.cc:348] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44dda970f9ff422591c49a6a679b280a" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39103 } } peers { permanent_uuid: "f385bc249b3d45e2ab894435ee55de85" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46821 } } peers { permanent_uuid: "c151ea3295ff4ad69955ac80e08ada4b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42397 } }
I20260810 23:32:22.089046 17057 raft_consensus.cc:374] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:22.089223 17057 raft_consensus.cc:729] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f385bc249b3d45e2ab894435ee55de85, State: Initialized, Role: FOLLOWER
I20260810 23:32:22.089598 17052 raft_consensus.cc:348] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44dda970f9ff422591c49a6a679b280a" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39103 } } peers { permanent_uuid: "f385bc249b3d45e2ab894435ee55de85" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46821 } } peers { permanent_uuid: "c151ea3295ff4ad69955ac80e08ada4b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42397 } }
I20260810 23:32:22.089919 17057 consensus_queue.cc:260] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44dda970f9ff422591c49a6a679b280a" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39103 } } peers { permanent_uuid: "f385bc249b3d45e2ab894435ee55de85" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46821 } } peers { permanent_uuid: "c151ea3295ff4ad69955ac80e08ada4b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42397 } }
I20260810 23:32:22.090108 17052 raft_consensus.cc:374] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:22.090205 17052 raft_consensus.cc:729] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c151ea3295ff4ad69955ac80e08ada4b, State: Initialized, Role: FOLLOWER
I20260810 23:32:22.090919 17052 consensus_queue.cc:260] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44dda970f9ff422591c49a6a679b280a" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39103 } } peers { permanent_uuid: "f385bc249b3d45e2ab894435ee55de85" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46821 } } peers { permanent_uuid: "c151ea3295ff4ad69955ac80e08ada4b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42397 } }
I20260810 23:32:22.091799 17057 ts_tablet_manager.cc:1428] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260810 23:32:22.092176 16804 heartbeater.cc:503] Master 127.9.48.254:40813 was elected leader, sending a full tablet report...
I20260810 23:32:22.093034 17052 ts_tablet_manager.cc:1428] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b: Time spent starting tablet: real 0.014s user 0.005s sys 0.008s
I20260810 23:32:22.098042 17055 raft_consensus.cc:348] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44dda970f9ff422591c49a6a679b280a" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39103 } } peers { permanent_uuid: "f385bc249b3d45e2ab894435ee55de85" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46821 } } peers { permanent_uuid: "c151ea3295ff4ad69955ac80e08ada4b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42397 } }
I20260810 23:32:22.099452 17055 raft_consensus.cc:374] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:22.099601 17055 raft_consensus.cc:729] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44dda970f9ff422591c49a6a679b280a, State: Initialized, Role: FOLLOWER
I20260810 23:32:22.100241 17055 consensus_queue.cc:260] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44dda970f9ff422591c49a6a679b280a" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39103 } } peers { permanent_uuid: "f385bc249b3d45e2ab894435ee55de85" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46821 } } peers { permanent_uuid: "c151ea3295ff4ad69955ac80e08ada4b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42397 } }
I20260810 23:32:22.101226 17016 heartbeater.cc:503] Master 127.9.48.254:40813 was elected leader, sending a full tablet report...
I20260810 23:32:22.101801 17055 ts_tablet_manager.cc:1428] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a: Time spent starting tablet: real 0.019s user 0.005s sys 0.010s
I20260810 23:32:22.163373 17063 raft_consensus.cc:482] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:22.163681 17063 raft_consensus.cc:504] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44dda970f9ff422591c49a6a679b280a" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39103 } } peers { permanent_uuid: "f385bc249b3d45e2ab894435ee55de85" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46821 } } peers { permanent_uuid: "c151ea3295ff4ad69955ac80e08ada4b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42397 } }
I20260810 23:32:22.165504 17063 leader_election.cc:288] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44dda970f9ff422591c49a6a679b280a (127.9.48.195:39103), f385bc249b3d45e2ab894435ee55de85 (127.9.48.193:46821)
I20260810 23:32:22.172684 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:22.165874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:60339 (local address 127.9.48.195:39103)
0810 23:32:22.166136 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:22.166143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:22.166422 (+ 279us) server_negotiation.cc:407] Connection header received
0810 23:32:22.166650 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:22.166657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:22.166749 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:22.166867 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:22.167534 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.168623 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.170665 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.171656 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.171775 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:22.171837 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:22.172245 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 23:32:22.172391 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 23:32:22.172791 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:22.165686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:39103 (local address 127.9.48.194:60339)
0810 23:32:22.166296 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:22.166323 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:22.166501 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:22.166891 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:22.166900 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:22.166933 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:22.167350 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:22.167364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.168750 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.168756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:22.170507 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:22.170519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.171869 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.171877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:22.172052 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:22.172088 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:32:22.172227 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 23:32:22.172369 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":303,"threads_started":1}
I20260810 23:32:22.173424 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:22.166719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:49509 (local address 127.9.48.193:46821)
0810 23:32:22.166904 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:22.166910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:22.167188 (+ 278us) server_negotiation.cc:407] Connection header received
0810 23:32:22.167315 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:22.167320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:22.167407 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:22.167499 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:22.168166 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.169181 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.171263 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.172213 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.172365 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:22.172430 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:22.173046 (+ 616us) server_negotiation.cc:299] Negotiation successful
0810 23:32:22.173184 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:32:22.174001 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:22.166427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46821 (local address 127.9.48.194:49509)
0810 23:32:22.167079 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:22.167093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:22.167197 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:22.167534 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:22.167539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:22.167564 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:22.167983 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:22.167997 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.169325 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.169329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:22.171102 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:22.171117 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.172585 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.172592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:22.172811 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:22.172845 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:32:22.173645 (+ 800us) client_negotiation.cc:240] Negotiation successful
0810 23:32:22.173733 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":477,"threads_started":1}
I20260810 23:32:22.173864 16950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bad9a5ea46a44dfd8ee6df69d8fc569d" candidate_uuid: "c151ea3295ff4ad69955ac80e08ada4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44dda970f9ff422591c49a6a679b280a" is_pre_election: true
I20260810 23:32:22.174444 16950 raft_consensus.cc:2454] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c151ea3295ff4ad69955ac80e08ada4b in term 0.
I20260810 23:32:22.175056 16512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bad9a5ea46a44dfd8ee6df69d8fc569d" candidate_uuid: "c151ea3295ff4ad69955ac80e08ada4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f385bc249b3d45e2ab894435ee55de85" is_pre_election: true
I20260810 23:32:22.175194 16619 leader_election.cc:302] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44dda970f9ff422591c49a6a679b280a, c151ea3295ff4ad69955ac80e08ada4b; no voters:
I20260810 23:32:22.175540 16512 raft_consensus.cc:2454] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c151ea3295ff4ad69955ac80e08ada4b in term 0.
I20260810 23:32:22.175635 17063 raft_consensus.cc:2781] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:22.175774 17063 raft_consensus.cc:482] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:22.175894 17063 raft_consensus.cc:3037] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:32:22.177939 16592 tablet.cc:2369] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:32:22.178377 17063 raft_consensus.cc:504] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44dda970f9ff422591c49a6a679b280a" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39103 } } peers { permanent_uuid: "f385bc249b3d45e2ab894435ee55de85" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46821 } } peers { permanent_uuid: "c151ea3295ff4ad69955ac80e08ada4b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42397 } }
I20260810 23:32:22.178982 17063 leader_election.cc:288] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [CANDIDATE]: Term 1 election: Requested vote from peers 44dda970f9ff422591c49a6a679b280a (127.9.48.195:39103), f385bc249b3d45e2ab894435ee55de85 (127.9.48.193:46821)
I20260810 23:32:22.179488 16950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bad9a5ea46a44dfd8ee6df69d8fc569d" candidate_uuid: "c151ea3295ff4ad69955ac80e08ada4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44dda970f9ff422591c49a6a679b280a"
I20260810 23:32:22.179555 16512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bad9a5ea46a44dfd8ee6df69d8fc569d" candidate_uuid: "c151ea3295ff4ad69955ac80e08ada4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f385bc249b3d45e2ab894435ee55de85"
I20260810 23:32:22.179661 16950 raft_consensus.cc:3037] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:22.179804 16512 raft_consensus.cc:3037] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:22.182075 16512 raft_consensus.cc:2454] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c151ea3295ff4ad69955ac80e08ada4b in term 1.
I20260810 23:32:22.182291 16950 raft_consensus.cc:2454] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c151ea3295ff4ad69955ac80e08ada4b in term 1.
I20260810 23:32:22.182518 16619 leader_election.cc:302] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c151ea3295ff4ad69955ac80e08ada4b, f385bc249b3d45e2ab894435ee55de85; no voters:
I20260810 23:32:22.182857 17063 raft_consensus.cc:2781] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:22.183298 17063 raft_consensus.cc:686] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [term 1 LEADER]: Becoming Leader. State: Replica: c151ea3295ff4ad69955ac80e08ada4b, State: Running, Role: LEADER
I20260810 23:32:22.183771 17063 consensus_queue.cc:237] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44dda970f9ff422591c49a6a679b280a" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39103 } } peers { permanent_uuid: "f385bc249b3d45e2ab894435ee55de85" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46821 } } peers { permanent_uuid: "c151ea3295ff4ad69955ac80e08ada4b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42397 } }
I20260810 23:32:22.188580 16272 catalog_manager.cc:5495] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b reported cstate change: term changed from 0 to 1, leader changed from <none> to c151ea3295ff4ad69955ac80e08ada4b (127.9.48.194). New cstate: current_term: 1 leader_uuid: "c151ea3295ff4ad69955ac80e08ada4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44dda970f9ff422591c49a6a679b280a" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f385bc249b3d45e2ab894435ee55de85" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c151ea3295ff4ad69955ac80e08ada4b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42397 } health_report { overall_health: HEALTHY } } }
W20260810 23:32:22.204325 16805 tablet.cc:2369] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:32:22.208209 17017 tablet.cc:2369] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:32:22.240409 17048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:22.235354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53268 (local address 127.9.48.194:42397)
0810 23:32:22.235575 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:22.235580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:22.235599 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:32:22.235819 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:22.235824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:22.235939 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:22.236065 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:32:22.236403 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.237204 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.238440 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.238844 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.238933 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:22.239352 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:32:22.239844 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:32:22.239965 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:22.240042 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 23:32:22.240103 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 23:32:22.248771 16512 raft_consensus.cc:1261] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85 [term 1 FOLLOWER]: Refusing update from remote peer c151ea3295ff4ad69955ac80e08ada4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:22.248967 16950 raft_consensus.cc:1261] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a [term 1 FOLLOWER]: Refusing update from remote peer c151ea3295ff4ad69955ac80e08ada4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:22.249831 17070 consensus_queue.cc:1035] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f385bc249b3d45e2ab894435ee55de85" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:22.250224 17063 consensus_queue.cc:1035] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "44dda970f9ff422591c49a6a679b280a" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:22.264905 17074 mvcc.cc:205] Tried to move back new op lower bound from 7317113824240357376 to 7317113823988625408. Current Snapshot: MvccSnapshot[applied={T|T < 7317113824240357376}]
I20260810 23:32:22.264919 17077 mvcc.cc:205] Tried to move back new op lower bound from 7317113824240357376 to 7317113823988625408. Current Snapshot: MvccSnapshot[applied={T|T < 7317113824240357376}]
I20260810 23:32:22.277979 16270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:22.278314 16270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:22.282269 16270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aeaa2874892446d1806026f3f15b4068: Sending DeleteTablet for 3 replicas of tablet bad9a5ea46a44dfd8ee6df69d8fc569d
I20260810 23:32:22.283464 16481 tablet_service.cc:1514] Processing DeleteTablet for tablet bad9a5ea46a44dfd8ee6df69d8fc569d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:22 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:44764
I20260810 23:32:22.283489 16688 tablet_service.cc:1514] Processing DeleteTablet for tablet bad9a5ea46a44dfd8ee6df69d8fc569d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:22 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:53258
I20260810 23:32:22.284385 16918 tablet_service.cc:1514] Processing DeleteTablet for tablet bad9a5ea46a44dfd8ee6df69d8fc569d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:22 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:60424
I20260810 23:32:22.285673 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 16377
I20260810 23:32:22.295847 17090 tablet_replica.cc:330] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a: stopping tablet replica
I20260810 23:32:22.296403 17090 raft_consensus.cc:2229] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:22.298278 17090 raft_consensus.cc:2258] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:22.301759 17090 ts_tablet_manager.cc:1905] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:22.304586 17086 tablet_replica.cc:330] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85: stopping tablet replica
I20260810 23:32:22.306794 17086 raft_consensus.cc:2229] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:22.307331 17086 raft_consensus.cc:2258] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:22.308213 17087 tablet_replica.cc:330] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b: stopping tablet replica
I20260810 23:32:22.308648 17087 raft_consensus.cc:2229] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:22.309291 17090 ts_tablet_manager.cc:1918] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:22.309435 17087 raft_consensus.cc:2258] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:22.309487 17090 log.cc:1196] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/bad9a5ea46a44dfd8ee6df69d8fc569d
I20260810 23:32:22.310099 17090 ts_tablet_manager.cc:1939] T bad9a5ea46a44dfd8ee6df69d8fc569d P 44dda970f9ff422591c49a6a679b280a: Deleting consensus metadata
I20260810 23:32:22.312134 16242 catalog_manager.cc:4841] TS 44dda970f9ff422591c49a6a679b280a (127.9.48.195:39103): tablet bad9a5ea46a44dfd8ee6df69d8fc569d (table test-table [id=e360a2e99b8b4a02a2f66a812ac871d8]) successfully deleted
I20260810 23:32:22.313347 17086 ts_tablet_manager.cc:1905] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 23:32:22 dist-test-slave-04bb krb5kdc[16202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:32:22.314375 17087 ts_tablet_manager.cc:1905] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:22.320865 17086 ts_tablet_manager.cc:1918] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:22.321055 17086 log.cc:1196] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/bad9a5ea46a44dfd8ee6df69d8fc569d
I20260810 23:32:22.321630 17086 ts_tablet_manager.cc:1939] T bad9a5ea46a44dfd8ee6df69d8fc569d P f385bc249b3d45e2ab894435ee55de85: Deleting consensus metadata
I20260810 23:32:22.322239 17087 ts_tablet_manager.cc:1918] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:22.322433 17087 log.cc:1196] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/bad9a5ea46a44dfd8ee6df69d8fc569d
I20260810 23:32:22.322594 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:22.287325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44774 (local address 127.9.48.193:46821)
0810 23:32:22.287550 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:22.287559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:22.298236 (+ 10677us) server_negotiation.cc:407] Connection header received
0810 23:32:22.298604 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:22.298609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:22.298754 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:22.298866 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:22.300948 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.302104 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.304764 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.305421 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.305525 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:22.314753 (+ 9228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:22.314769 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:22.314785 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:22.314838 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:22.318888 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:22.319773 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:22.319783 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:22.319787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:22.319973 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:22.320425 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:22.320430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:22.320434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:22.320961 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:22.321336 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:22.321978 (+ 642us) server_negotiation.cc:299] Negotiation successful
0810 23:32:22.322199 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:32:22.323179 17087 ts_tablet_manager.cc:1939] T bad9a5ea46a44dfd8ee6df69d8fc569d P c151ea3295ff4ad69955ac80e08ada4b: Deleting consensus metadata
I20260810 23:32:22.323523 16242 catalog_manager.cc:4841] TS f385bc249b3d45e2ab894435ee55de85 (127.9.48.193:46821): tablet bad9a5ea46a44dfd8ee6df69d8fc569d (table test-table [id=e360a2e99b8b4a02a2f66a812ac871d8]) successfully deleted
I20260810 23:32:22.324035 16581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:22.325394 16244 catalog_manager.cc:4841] TS c151ea3295ff4ad69955ac80e08ada4b (127.9.48.194:42397): tablet bad9a5ea46a44dfd8ee6df69d8fc569d (table test-table [id=e360a2e99b8b4a02a2f66a812ac871d8]) successfully deleted
I20260810 23:32:22.422969 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 16377
I20260810 23:32:22.451210 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 16599
Aug 10 23:32:22 dist-test-slave-04bb krb5kdc[16202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:32:22.472254 17048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:22.452586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.9.48.194:42397)
0810 23:32:22.452766 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:22.452773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:22.452794 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:32:22.452875 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:22.452880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:22.452980 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:22.453183 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:22.454776 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.456066 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.457997 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.458719 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.458855 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:22.464333 (+ 5478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:22.464346 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:22.464363 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:22.464406 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:22.468691 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:22.469567 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:22.469576 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:22.469581 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:22.469788 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:22.470250 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:22.470259 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:22.470263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:22.470703 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:22.471414 (+ 711us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:22.471602 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 23:32:22.471837 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:32:22.473184 16797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:22.588277 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 16599
I20260810 23:32:22.615700 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 16811
Aug 10 23:32:22 dist-test-slave-04bb krb5kdc[16202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:32:22.630587 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:22.616524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60438 (local address 127.9.48.195:39103)
0810 23:32:22.616725 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:22.616730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:22.616763 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:32:22.616961 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:22.616965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:22.617043 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:22.617127 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:22.618350 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.619056 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.620367 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:22.620860 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:22.620956 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:22.625356 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:22.625369 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:22.625384 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:22.625415 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:22.628058 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:22.628746 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:22.628754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:22.628756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:22.628891 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:22.629246 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:22.629249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:22.629251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:22.629600 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:22.629836 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:22.630060 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 23:32:22.630227 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:32:22.631232 17009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:22.719447 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 16811
I20260810 23:32:22.746524 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 16220
I20260810 23:32:22.747748 16321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:22.858495 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 16220
2026-08-10T23:32:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4541 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T23:32:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:22Z Disabled control of system clock
I20260810 23:32:22.943656 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:41409
--webserver_interface=127.9.48.254
--webserver_port=0
--builtin_ntp_servers=127.9.48.212:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:41409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 23:32:23.272753 17104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:23.273026 17104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:23.273113 17104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:23.280103 17104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:23.280207 17104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:23.280247 17104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:23.280296 17104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:23.280339 17104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:23.289889 17104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:41409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:41409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:23.291495 17104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:23.293401 17104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:23.305630 17111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:23.305599 17110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:23.305780 17113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:23.306886 17104 server_base.cc:1034] running on GCE node
I20260810 23:32:23.307667 17104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:23.309212 17104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:23.310444 17104 hybrid_clock.cc:648] HybridClock initialized: now 1786404743310384 us; error 57 us; skew 500 ppm
I20260810 23:32:23.310792 17104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:32:23.313093 17104 webserver.cc:460] Webserver started at http://127.9.48.254:34939/ using document root <none> and password file <none>
I20260810 23:32:23.313788 17104 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:23.313926 17104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:23.314257 17104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:23.317183 17104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "aebe8e4e14314e5f8b90dd9f1cf40fd3"
format_stamp: "Formatted at 2026-08-10 23:32:23 on dist-test-slave-04bb"
server_key: "d63f030798ed8816a98fe56578331d3b"
server_key_iv: "c274db0b01f68548ffd13c578f008fe3"
server_key_version: "encryptionkey@0"
I20260810 23:32:23.317982 17104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "aebe8e4e14314e5f8b90dd9f1cf40fd3"
format_stamp: "Formatted at 2026-08-10 23:32:23 on dist-test-slave-04bb"
server_key: "d63f030798ed8816a98fe56578331d3b"
server_key_iv: "c274db0b01f68548ffd13c578f008fe3"
server_key_version: "encryptionkey@0"
I20260810 23:32:23.322961 17104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:32:23.326395 17121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:23.327674 17104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:32:23.327870 17104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "aebe8e4e14314e5f8b90dd9f1cf40fd3"
format_stamp: "Formatted at 2026-08-10 23:32:23 on dist-test-slave-04bb"
server_key: "d63f030798ed8816a98fe56578331d3b"
server_key_iv: "c274db0b01f68548ffd13c578f008fe3"
server_key_version: "encryptionkey@0"
I20260810 23:32:23.328084 17104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:23.333745 17104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:23.334477 17104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:23.334779 17104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:23.353602 17104 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:41409
I20260810 23:32:23.353685 17206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:41409 every 8 connection(s)
I20260810 23:32:23.354898 17104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:23.358724 17207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:23.362465 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 17104
I20260810 23:32:23.362663 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:32:23.363384 9411 external_mini_cluster.cc:1428] Setting key fc15292db2c7a23c83a5cf4f52193711
I20260810 23:32:23.371569 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:23.365416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48738 (local address 127.9.48.254:41409)
0810 23:32:23.366350 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:23.366360 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:23.366402 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:32:23.367913 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:23.367934 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:23.368366 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:23.368746 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:23.369996 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:23.370003 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:23.370018 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:23.370036 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:23.370096 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:23.370113 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:23.370308 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:23.370450 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 23:32:23.370690 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":334,"threads_started":1}
I20260810 23:32:23.372429 17207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3: Bootstrap starting.
I20260810 23:32:23.375917 17207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:23.377182 17207 log.cc:824] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:23.379848 17207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3: No bootstrap required, opened a new log
I20260810 23:32:23.387620 17207 raft_consensus.cc:348] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebe8e4e14314e5f8b90dd9f1cf40fd3" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41409 } }
I20260810 23:32:23.387986 17207 raft_consensus.cc:374] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:23.388072 17207 raft_consensus.cc:729] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aebe8e4e14314e5f8b90dd9f1cf40fd3, State: Initialized, Role: FOLLOWER
I20260810 23:32:23.388796 17207 consensus_queue.cc:260] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebe8e4e14314e5f8b90dd9f1cf40fd3" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41409 } }
I20260810 23:32:23.389129 17207 raft_consensus.cc:388] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:32:23.389285 17207 raft_consensus.cc:482] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:32:23.389500 17207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:23.391847 17207 raft_consensus.cc:504] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebe8e4e14314e5f8b90dd9f1cf40fd3" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41409 } }
I20260810 23:32:23.392400 17207 leader_election.cc:302] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aebe8e4e14314e5f8b90dd9f1cf40fd3; no voters:
I20260810 23:32:23.393052 17207 leader_election.cc:288] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:32:23.393170 17213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:23.394105 17213 raft_consensus.cc:686] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [term 1 LEADER]: Becoming Leader. State: Replica: aebe8e4e14314e5f8b90dd9f1cf40fd3, State: Running, Role: LEADER
I20260810 23:32:23.394799 17213 consensus_queue.cc:237] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebe8e4e14314e5f8b90dd9f1cf40fd3" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41409 } }
I20260810 23:32:23.395597 17207 sys_catalog.cc:564] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:23.397877 17215 sys_catalog.cc:455] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aebe8e4e14314e5f8b90dd9f1cf40fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebe8e4e14314e5f8b90dd9f1cf40fd3" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41409 } } }
I20260810 23:32:23.398350 17215 sys_catalog.cc:458] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:23.398293 17216 sys_catalog.cc:455] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aebe8e4e14314e5f8b90dd9f1cf40fd3. Latest consensus state: current_term: 1 leader_uuid: "aebe8e4e14314e5f8b90dd9f1cf40fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aebe8e4e14314e5f8b90dd9f1cf40fd3" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41409 } } }
I20260810 23:32:23.398607 17216 sys_catalog.cc:458] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:23.400107 17221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:23.405745 17221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:23.415241 17221 catalog_manager.cc:1347] Generated new cluster ID: a9f92367b69c4033839c4460a9696789
I20260810 23:32:23.415378 17221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:23.443322 17221 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:32:23.443821 17221 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:23.453336 17221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3: Generated new TSK 0
I20260810 23:32:23.454203 17221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:32:23.471020 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41409
--builtin_ntp_servers=127.9.48.212:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 23:32:23.833138 17242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:23.833439 17242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:23.833636 17242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:23.841136 17242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:23.841243 17242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:23.841462 17242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:32:23.850689 17242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:23.852025 17242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:23.854001 17242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:23.867830 17249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:23.867878 17252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:23.867926 17250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:23.868664 17242 server_base.cc:1034] running on GCE node
I20260810 23:32:23.869311 17242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:23.870296 17242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:23.871536 17242 hybrid_clock.cc:648] HybridClock initialized: now 1786404743871489 us; error 69 us; skew 500 ppm
I20260810 23:32:23.871855 17242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:32:23.874137 17242 webserver.cc:460] Webserver started at http://127.9.48.193:45161/ using document root <none> and password file <none>
I20260810 23:32:23.874858 17242 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:23.874995 17242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:23.875325 17242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:23.878403 17242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "da031898cdc242a98d5db97a361de40c"
format_stamp: "Formatted at 2026-08-10 23:32:23 on dist-test-slave-04bb"
server_key: "40ede3aefebd2d50613b7bc63f07a9cb"
server_key_iv: "3186cacecd06ce0053395a3162810bcb"
server_key_version: "encryptionkey@0"
I20260810 23:32:23.879225 17242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "da031898cdc242a98d5db97a361de40c"
format_stamp: "Formatted at 2026-08-10 23:32:23 on dist-test-slave-04bb"
server_key: "40ede3aefebd2d50613b7bc63f07a9cb"
server_key_iv: "3186cacecd06ce0053395a3162810bcb"
server_key_version: "encryptionkey@0"
I20260810 23:32:23.884483 17242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 23:32:23.888068 17261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:23.889494 17242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:23.889631 17242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "da031898cdc242a98d5db97a361de40c"
format_stamp: "Formatted at 2026-08-10 23:32:23 on dist-test-slave-04bb"
server_key: "40ede3aefebd2d50613b7bc63f07a9cb"
server_key_iv: "3186cacecd06ce0053395a3162810bcb"
server_key_version: "encryptionkey@0"
I20260810 23:32:23.889829 17242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:23.895462 17242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:23.896317 17242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:23.896647 17242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:23.897675 17242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:23.899127 17242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:23.899271 17242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:23.899421 17242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:23.899507 17242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:23.931445 17242 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:45579
I20260810 23:32:23.931557 17448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:45579 every 8 connection(s)
I20260810 23:32:23.933043 17242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:32:23.939319 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 17242
I20260810 23:32:23.939534 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:32:23.940308 9411 external_mini_cluster.cc:1428] Setting key 6ac7c984d497077a4b1151ec152d83e1
I20260810 23:32:23.941740 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:23.935813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:45851 (local address 127.9.48.254:41409)
0810 23:32:23.936297 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:23.936302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:23.937442 (+ 1140us) server_negotiation.cc:407] Connection header received
0810 23:32:23.939167 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:23.939174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:23.939251 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:23.939373 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:23.940915 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:23.940921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:23.940924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:23.940940 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:23.940984 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:23.940996 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:23.941139 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:23.941342 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 23:32:23.941414 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":271,"threads_started":1}
I20260810 23:32:23.942631 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:23.936326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:41409 (local address 127.9.48.193:45851)
0810 23:32:23.937260 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:23.937297 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:23.938889 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:23.939650 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:23.939661 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:23.939771 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:23.940599 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:32:23.940601 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:23.940641 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:32:23.940728 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:23.941100 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:23.941114 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:23.941123 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:32:23.941430 (+ 307us) client_negotiation.cc:240] Negotiation successful
0810 23:32:23.941677 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":320,"threads_started":1}
I20260810 23:32:23.943408 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41409
--builtin_ntp_servers=127.9.48.212:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 23:32:23.944182 17449 heartbeater.cc:347] Connected to a master server at 127.9.48.254:41409
I20260810 23:32:23.944478 17449 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:23.945058 17449 heartbeater.cc:511] Master 127.9.48.254:41409 requested a full tablet report, sending...
I20260810 23:32:23.947268 17151 ts_manager.cc:194] Registered new tserver with Master: da031898cdc242a98d5db97a361de40c (127.9.48.193:45579)
W20260810 23:32:24.276810 17457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:24.277067 17457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:24.277158 17457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:24.284174 17457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:24.284263 17457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:24.284370 17457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:32:24.294737 17457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:24.296155 17457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:24.298256 17457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:24.312283 17466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:24.312721 17467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:24.313438 17469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:24.313753 17457 server_base.cc:1034] running on GCE node
I20260810 23:32:24.314481 17457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:24.315557 17457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:24.316793 17457 hybrid_clock.cc:648] HybridClock initialized: now 1786404744316746 us; error 39 us; skew 500 ppm
I20260810 23:32:24.317143 17457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:32:24.319696 17457 webserver.cc:460] Webserver started at http://127.9.48.194:42979/ using document root <none> and password file <none>
I20260810 23:32:24.320439 17457 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:24.320616 17457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:24.320951 17457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:24.324283 17457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "53cf4a6397aa43afafa769c9e92f9618"
format_stamp: "Formatted at 2026-08-10 23:32:24 on dist-test-slave-04bb"
server_key: "6f605475200a051551e9b7325c7673f9"
server_key_iv: "5bf296237057931d64c9468336657d3d"
server_key_version: "encryptionkey@0"
I20260810 23:32:24.325138 17457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "53cf4a6397aa43afafa769c9e92f9618"
format_stamp: "Formatted at 2026-08-10 23:32:24 on dist-test-slave-04bb"
server_key: "6f605475200a051551e9b7325c7673f9"
server_key_iv: "5bf296237057931d64c9468336657d3d"
server_key_version: "encryptionkey@0"
I20260810 23:32:24.330210 17457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:32:24.333462 17478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:24.334736 17457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:32:24.334867 17457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "53cf4a6397aa43afafa769c9e92f9618"
format_stamp: "Formatted at 2026-08-10 23:32:24 on dist-test-slave-04bb"
server_key: "6f605475200a051551e9b7325c7673f9"
server_key_iv: "5bf296237057931d64c9468336657d3d"
server_key_version: "encryptionkey@0"
I20260810 23:32:24.335053 17457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:24.340667 17457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:24.341451 17457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:24.341701 17457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:24.342530 17457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:24.343803 17457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:24.343896 17457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:24.343974 17457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:24.344027 17457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:24.380600 17457 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:38131
I20260810 23:32:24.380698 17652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:38131 every 8 connection(s)
I20260810 23:32:24.381907 17457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:32:24.388890 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 17457
I20260810 23:32:24.389086 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:32:24.389734 9411 external_mini_cluster.cc:1428] Setting key 454a7e5f0a202f3f7bc39d18765c59d3
I20260810 23:32:24.390257 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:24.384450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:54993 (local address 127.9.48.254:41409)
0810 23:32:24.384647 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:24.384652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:24.386041 (+ 1389us) server_negotiation.cc:407] Connection header received
0810 23:32:24.387772 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:24.387778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:24.387866 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:24.387989 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:24.389565 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:24.389570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:24.389574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:24.389590 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:24.389623 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:24.389635 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:24.389738 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:24.389993 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 23:32:24.390050 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:32:24.391654 17659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:24.384960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:41409 (local address 127.9.48.194:54993)
0810 23:32:24.385876 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:24.385923 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:24.387491 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:24.388225 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:24.388232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:24.388325 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:24.389199 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:32:24.389203 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:24.389255 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:32:24.389369 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:24.389752 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:24.389764 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:24.389775 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 23:32:24.390077 (+ 302us) client_negotiation.cc:240] Negotiation successful
0810 23:32:24.390643 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":313,"threads_started":1}
I20260810 23:32:24.392715 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41409
--builtin_ntp_servers=127.9.48.212:42427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 23:32:24.393070 17657 heartbeater.cc:347] Connected to a master server at 127.9.48.254:41409
I20260810 23:32:24.393383 17657 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:24.394093 17657 heartbeater.cc:511] Master 127.9.48.254:41409 requested a full tablet report, sending...
I20260810 23:32:24.395610 17151 ts_manager.cc:194] Registered new tserver with Master: 53cf4a6397aa43afafa769c9e92f9618 (127.9.48.194:38131)
W20260810 23:32:24.728070 17660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:24.728391 17660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:24.728590 17660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:24.735939 17660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:24.736068 17660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:24.736258 17660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:32:24.745342 17660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:42427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:24.746613 17660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:24.748644 17660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:24.762141 17670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:24.762171 17671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:24.762991 17673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:24.763522 17660 server_base.cc:1034] running on GCE node
I20260810 23:32:24.764055 17660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:24.765039 17660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:24.766279 17660 hybrid_clock.cc:648] HybridClock initialized: now 1786404744766226 us; error 52 us; skew 500 ppm
I20260810 23:32:24.766597 17660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:32:24.768908 17660 webserver.cc:460] Webserver started at http://127.9.48.195:36237/ using document root <none> and password file <none>
I20260810 23:32:24.769603 17660 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:24.769747 17660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:24.770085 17660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:24.773008 17660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "e50b47491d3240818e301fbfc3ce363b"
format_stamp: "Formatted at 2026-08-10 23:32:24 on dist-test-slave-04bb"
server_key: "94956538ee0daa0502aa8fed4cc3149e"
server_key_iv: "568de2715f07900a37cd5aa87a516a51"
server_key_version: "encryptionkey@0"
I20260810 23:32:24.773813 17660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "e50b47491d3240818e301fbfc3ce363b"
format_stamp: "Formatted at 2026-08-10 23:32:24 on dist-test-slave-04bb"
server_key: "94956538ee0daa0502aa8fed4cc3149e"
server_key_iv: "568de2715f07900a37cd5aa87a516a51"
server_key_version: "encryptionkey@0"
I20260810 23:32:24.779095 17660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 23:32:24.782531 17681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:24.783823 17660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260810 23:32:24.784001 17660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "e50b47491d3240818e301fbfc3ce363b"
format_stamp: "Formatted at 2026-08-10 23:32:24 on dist-test-slave-04bb"
server_key: "94956538ee0daa0502aa8fed4cc3149e"
server_key_iv: "568de2715f07900a37cd5aa87a516a51"
server_key_version: "encryptionkey@0"
I20260810 23:32:24.784246 17660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:24.790055 17660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:24.790831 17660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:24.791112 17660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:24.792052 17660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:24.793685 17660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:24.793812 17660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:24.793942 17660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:24.794040 17660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:24.829103 17660 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:45705
I20260810 23:32:24.829257 17838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:45705 every 8 connection(s)
I20260810 23:32:24.830451 17660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:32:24.836468 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 17660
I20260810 23:32:24.836694 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:32:24.837378 9411 external_mini_cluster.cc:1428] Setting key bebf4f12c427802f2880a5c766e93eb4
I20260810 23:32:24.840804 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:24.833504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:42025 (local address 127.9.48.254:41409)
0810 23:32:24.833729 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:24.833735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:24.835260 (+ 1525us) server_negotiation.cc:407] Connection header received
0810 23:32:24.837958 (+ 2698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:24.837964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:24.838059 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:24.838235 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:24.839923 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:24.839930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:24.839935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:24.839956 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:24.840000 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:24.840014 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:24.840167 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:24.840377 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 23:32:24.840452 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 23:32:24.841879 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:24.834078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:41409 (local address 127.9.48.195:42025)
0810 23:32:24.835071 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:24.835133 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:24.837562 (+ 2429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:24.838523 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:24.838570 (+ 47us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:24.838695 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:24.839523 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:32:24.839526 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:24.839579 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:32:24.839734 (+ 155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:24.840130 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:24.840142 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:24.840150 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:32:24.840477 (+ 327us) client_negotiation.cc:240] Negotiation successful
0810 23:32:24.840780 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":303,"threads_started":1}
I20260810 23:32:24.843217 17839 heartbeater.cc:347] Connected to a master server at 127.9.48.254:41409
I20260810 23:32:24.843506 17839 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:24.844069 17839 heartbeater.cc:511] Master 127.9.48.254:41409 requested a full tablet report, sending...
I20260810 23:32:24.845430 17151 ts_manager.cc:194] Registered new tserver with Master: e50b47491d3240818e301fbfc3ce363b (127.9.48.195:45705)
I20260810 23:32:24.854151 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:32:24.862000 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:24.859385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.9.48.254:41409)
0810 23:32:24.859562 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:24.859567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:24.860176 (+ 609us) server_negotiation.cc:407] Connection header received
0810 23:32:24.860297 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:24.860301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:24.860355 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:24.860441 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:24.861489 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:24.861495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:24.861498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:24.861516 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:24.861554 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:24.861566 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:24.861686 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:24.861743 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 23:32:24.861791 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:32:24.866537 17151 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:24.873463 17151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:24.891433 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:24.885736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:45579 (local address 127.0.0.1:43298)
0810 23:32:24.886910 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:24.886960 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:24.887152 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:24.888599 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:24.888606 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:24.888707 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:24.889663 (+ 956us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:32:24.889667 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:24.889705 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:32:24.889792 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:24.890464 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:24.890475 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:24.890483 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:32:24.890706 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 23:32:24.890818 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":2604,"threads_started":1}
I20260810 23:32:24.891784 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:24.886682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.9.48.195:45705)
0810 23:32:24.888380 (+ 1698us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:24.888390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:24.888936 (+ 546us) server_negotiation.cc:407] Connection header received
0810 23:32:24.889018 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:24.889028 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:24.889393 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:24.889550 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:24.890928 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:24.890943 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:24.890954 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:24.890969 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:24.891021 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:24.891036 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:24.891220 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:24.891299 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 23:32:24.891397 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1500,"thread_start_us":1328,"threads_started":1}
I20260810 23:32:24.892057 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:24.886005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.9.48.193:45579)
0810 23:32:24.887883 (+ 1878us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:24.887895 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:24.887933 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:32:24.888040 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:24.888051 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:24.888359 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:24.888928 (+ 569us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:24.890101 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:24.890113 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:24.890132 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:24.890161 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:24.890226 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:24.890251 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:24.890410 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:24.891565 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0810 23:32:24.891672 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1653,"thread_start_us":1488,"threads_started":1}
I20260810 23:32:24.892800 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:24.886038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:45705 (local address 127.0.0.1:36784)
0810 23:32:24.888662 (+ 2624us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:24.888681 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:24.888791 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:24.889989 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:24.889996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:24.890024 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:24.890723 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:32:24.890726 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:24.890739 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:32:24.890774 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:24.891154 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:24.891160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:24.891168 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:32:24.891326 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 23:32:24.891381 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2461,"thread_start_us":843,"threads_started":1}
I20260810 23:32:24.895714 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:24.890450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:38131 (local address 127.0.0.1:58120)
0810 23:32:24.891978 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:24.891999 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:24.892089 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:24.893942 (+ 1853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:24.893947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:24.893983 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:24.894855 (+ 872us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:32:24.894858 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:24.894870 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:32:24.894903 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:24.895310 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:24.895314 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:24.895319 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:32:24.895407 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:32:24.895455 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1402,"thread_start_us":2128,"threads_started":1}
I20260810 23:32:24.895833 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:24.889031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.9.48.194:38131)
0810 23:32:24.893119 (+ 4088us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:24.893127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:24.893157 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:32:24.893228 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:24.893237 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:24.893440 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:24.893585 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:24.895067 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:24.895074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:24.895083 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:24.895098 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:24.895149 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:24.895167 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:24.895355 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:24.895434 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 23:32:24.895532 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3923,"thread_start_us":3853,"threads_started":1}
I20260810 23:32:24.896970 17750 tablet_service.cc:1467] Processing CreateTablet for tablet 8500518e0c924e6992b4e8d42cdeea9b (DEFAULT_TABLE table=test-table [id=326b801c46514a3cbd100804caba9e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.896970 17344 tablet_service.cc:1467] Processing CreateTablet for tablet 8500518e0c924e6992b4e8d42cdeea9b (DEFAULT_TABLE table=test-table [id=326b801c46514a3cbd100804caba9e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.898569 17344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8500518e0c924e6992b4e8d42cdeea9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.898568 17750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8500518e0c924e6992b4e8d42cdeea9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.899441 17555 tablet_service.cc:1467] Processing CreateTablet for tablet 8500518e0c924e6992b4e8d42cdeea9b (DEFAULT_TABLE table=test-table [id=326b801c46514a3cbd100804caba9e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:24.900842 17555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8500518e0c924e6992b4e8d42cdeea9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:24.913059 17874 tablet_bootstrap.cc:492] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618: Bootstrap starting.
I20260810 23:32:24.917160 17874 tablet_bootstrap.cc:654] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:24.918205 17874 log.cc:824] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:24.920717 17879 tablet_bootstrap.cc:492] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b: Bootstrap starting.
I20260810 23:32:24.921404 17874 tablet_bootstrap.cc:492] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618: No bootstrap required, opened a new log
I20260810 23:32:24.921687 17874 ts_tablet_manager.cc:1397] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260810 23:32:24.923877 17879 tablet_bootstrap.cc:654] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:24.924866 17879 log.cc:824] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:24.925555 17876 tablet_bootstrap.cc:492] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c: Bootstrap starting.
I20260810 23:32:24.927986 17879 tablet_bootstrap.cc:492] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b: No bootstrap required, opened a new log
I20260810 23:32:24.928268 17879 ts_tablet_manager.cc:1397] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 23:32:24.929109 17876 tablet_bootstrap.cc:654] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:24.929673 17874 raft_consensus.cc:348] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da031898cdc242a98d5db97a361de40c" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 45579 } } peers { permanent_uuid: "e50b47491d3240818e301fbfc3ce363b" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45705 } } peers { permanent_uuid: "53cf4a6397aa43afafa769c9e92f9618" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38131 } }
I20260810 23:32:24.930047 17874 raft_consensus.cc:374] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:24.930179 17874 raft_consensus.cc:729] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53cf4a6397aa43afafa769c9e92f9618, State: Initialized, Role: FOLLOWER
I20260810 23:32:24.930334 17876 log.cc:824] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:24.930975 17874 consensus_queue.cc:260] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da031898cdc242a98d5db97a361de40c" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 45579 } } peers { permanent_uuid: "e50b47491d3240818e301fbfc3ce363b" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45705 } } peers { permanent_uuid: "53cf4a6397aa43afafa769c9e92f9618" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38131 } }
I20260810 23:32:24.940955 17879 raft_consensus.cc:348] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da031898cdc242a98d5db97a361de40c" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 45579 } } peers { permanent_uuid: "e50b47491d3240818e301fbfc3ce363b" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45705 } } peers { permanent_uuid: "53cf4a6397aa43afafa769c9e92f9618" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38131 } }
I20260810 23:32:24.941413 17879 raft_consensus.cc:374] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:24.941453 17876 tablet_bootstrap.cc:492] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c: No bootstrap required, opened a new log
I20260810 23:32:24.941536 17879 raft_consensus.cc:729] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50b47491d3240818e301fbfc3ce363b, State: Initialized, Role: FOLLOWER
I20260810 23:32:24.941866 17876 ts_tablet_manager.cc:1397] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.004s
I20260810 23:32:24.942157 17879 consensus_queue.cc:260] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da031898cdc242a98d5db97a361de40c" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 45579 } } peers { permanent_uuid: "e50b47491d3240818e301fbfc3ce363b" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45705 } } peers { permanent_uuid: "53cf4a6397aa43afafa769c9e92f9618" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38131 } }
I20260810 23:32:24.949389 17657 heartbeater.cc:503] Master 127.9.48.254:41409 was elected leader, sending a full tablet report...
I20260810 23:32:24.950310 17449 heartbeater.cc:503] Master 127.9.48.254:41409 was elected leader, sending a full tablet report...
I20260810 23:32:24.953745 17874 ts_tablet_manager.cc:1428] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618: Time spent starting tablet: real 0.032s user 0.015s sys 0.012s
I20260810 23:32:24.955915 17839 heartbeater.cc:503] Master 127.9.48.254:41409 was elected leader, sending a full tablet report...
I20260810 23:32:24.956828 17879 ts_tablet_manager.cc:1428] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b: Time spent starting tablet: real 0.028s user 0.014s sys 0.016s
I20260810 23:32:24.957329 17876 raft_consensus.cc:348] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da031898cdc242a98d5db97a361de40c" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 45579 } } peers { permanent_uuid: "e50b47491d3240818e301fbfc3ce363b" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45705 } } peers { permanent_uuid: "53cf4a6397aa43afafa769c9e92f9618" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38131 } }
I20260810 23:32:24.957791 17876 raft_consensus.cc:374] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:24.957914 17876 raft_consensus.cc:729] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da031898cdc242a98d5db97a361de40c, State: Initialized, Role: FOLLOWER
I20260810 23:32:24.958623 17876 consensus_queue.cc:260] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da031898cdc242a98d5db97a361de40c" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 45579 } } peers { permanent_uuid: "e50b47491d3240818e301fbfc3ce363b" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45705 } } peers { permanent_uuid: "53cf4a6397aa43afafa769c9e92f9618" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38131 } }
I20260810 23:32:24.960281 17876 ts_tablet_manager.cc:1428] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c: Time spent starting tablet: real 0.018s user 0.014s sys 0.001s
I20260810 23:32:25.080194 17891 raft_consensus.cc:482] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:25.080605 17891 raft_consensus.cc:504] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da031898cdc242a98d5db97a361de40c" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 45579 } } peers { permanent_uuid: "e50b47491d3240818e301fbfc3ce363b" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45705 } } peers { permanent_uuid: "53cf4a6397aa43afafa769c9e92f9618" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38131 } }
W20260810 23:32:25.082226 17841 tablet.cc:2369] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:32:25.082476 17891 leader_election.cc:288] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e50b47491d3240818e301fbfc3ce363b (127.9.48.195:45705), 53cf4a6397aa43afafa769c9e92f9618 (127.9.48.194:38131)
I20260810 23:32:25.086350 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:25.083517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:50697 (local address 127.9.48.195:45705)
0810 23:32:25.083713 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:25.083717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:25.084201 (+ 484us) server_negotiation.cc:407] Connection header received
0810 23:32:25.084421 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:25.084426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:25.084530 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:25.084707 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:25.085721 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:25.085728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:25.085737 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:25.085759 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:25.085787 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:25.085798 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:25.085934 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:25.086011 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 23:32:25.086078 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:32:25.086599 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:25.082951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:45705 (local address 127.9.48.193:50697)
0810 23:32:25.084092 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:25.084107 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:25.084245 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:25.084736 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:25.084740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:25.084764 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:25.085517 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:32:25.085520 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:25.085535 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:32:25.085573 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:25.085899 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:25.085903 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:25.085910 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:32:25.086048 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 23:32:25.086099 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"thread_start_us":814,"threads_started":1}
I20260810 23:32:25.086889 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:25.083071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:36209 (local address 127.9.48.194:38131)
0810 23:32:25.083261 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:25.083270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:25.084052 (+ 782us) server_negotiation.cc:407] Connection header received
0810 23:32:25.084311 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:25.084320 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:25.084393 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:25.084471 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:25.086015 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:25.086027 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:25.086031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:25.086054 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:25.086127 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:25.086143 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:25.086303 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:25.086443 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 23:32:25.086522 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:32:25.087091 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:25.082762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:38131 (local address 127.9.48.193:36209)
0810 23:32:25.083924 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:25.083943 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:25.084111 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:25.084617 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:25.084627 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:25.084663 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:25.085763 (+ 1100us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:32:25.085766 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:25.085782 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:32:25.085826 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:25.086268 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:25.086274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:25.086280 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:32:25.086514 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 23:32:25.086590 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"thread_start_us":676,"threads_started":1}
I20260810 23:32:25.087651 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500518e0c924e6992b4e8d42cdeea9b" candidate_uuid: "da031898cdc242a98d5db97a361de40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50b47491d3240818e301fbfc3ce363b" is_pre_election: true
I20260810 23:32:25.088256 17781 raft_consensus.cc:2454] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da031898cdc242a98d5db97a361de40c in term 0.
I20260810 23:32:25.088505 17581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500518e0c924e6992b4e8d42cdeea9b" candidate_uuid: "da031898cdc242a98d5db97a361de40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53cf4a6397aa43afafa769c9e92f9618" is_pre_election: true
I20260810 23:32:25.089114 17581 raft_consensus.cc:2454] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da031898cdc242a98d5db97a361de40c in term 0.
I20260810 23:32:25.089151 17268 leader_election.cc:302] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da031898cdc242a98d5db97a361de40c, e50b47491d3240818e301fbfc3ce363b; no voters:
I20260810 23:32:25.089596 17891 raft_consensus.cc:2781] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:25.089813 17891 raft_consensus.cc:482] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:25.089974 17891 raft_consensus.cc:3037] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:25.092746 17891 raft_consensus.cc:504] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da031898cdc242a98d5db97a361de40c" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 45579 } } peers { permanent_uuid: "e50b47491d3240818e301fbfc3ce363b" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45705 } } peers { permanent_uuid: "53cf4a6397aa43afafa769c9e92f9618" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38131 } }
I20260810 23:32:25.093648 17891 leader_election.cc:288] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [CANDIDATE]: Term 1 election: Requested vote from peers e50b47491d3240818e301fbfc3ce363b (127.9.48.195:45705), 53cf4a6397aa43afafa769c9e92f9618 (127.9.48.194:38131)
I20260810 23:32:25.094031 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500518e0c924e6992b4e8d42cdeea9b" candidate_uuid: "da031898cdc242a98d5db97a361de40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e50b47491d3240818e301fbfc3ce363b"
I20260810 23:32:25.094177 17581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8500518e0c924e6992b4e8d42cdeea9b" candidate_uuid: "da031898cdc242a98d5db97a361de40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53cf4a6397aa43afafa769c9e92f9618"
I20260810 23:32:25.094283 17781 raft_consensus.cc:3037] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:25.094394 17581 raft_consensus.cc:3037] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:25.096992 17581 raft_consensus.cc:2454] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da031898cdc242a98d5db97a361de40c in term 1.
I20260810 23:32:25.097398 17781 raft_consensus.cc:2454] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da031898cdc242a98d5db97a361de40c in term 1.
I20260810 23:32:25.097424 17267 leader_election.cc:302] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53cf4a6397aa43afafa769c9e92f9618, da031898cdc242a98d5db97a361de40c; no voters:
I20260810 23:32:25.097731 17891 raft_consensus.cc:2781] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:25.098237 17891 raft_consensus.cc:686] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [term 1 LEADER]: Becoming Leader. State: Replica: da031898cdc242a98d5db97a361de40c, State: Running, Role: LEADER
I20260810 23:32:25.098802 17891 consensus_queue.cc:237] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da031898cdc242a98d5db97a361de40c" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 45579 } } peers { permanent_uuid: "e50b47491d3240818e301fbfc3ce363b" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45705 } } peers { permanent_uuid: "53cf4a6397aa43afafa769c9e92f9618" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38131 } }
I20260810 23:32:25.104208 17151 catalog_manager.cc:5495] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c reported cstate change: term changed from 0 to 1, leader changed from <none> to da031898cdc242a98d5db97a361de40c (127.9.48.193). New cstate: current_term: 1 leader_uuid: "da031898cdc242a98d5db97a361de40c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da031898cdc242a98d5db97a361de40c" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 45579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e50b47491d3240818e301fbfc3ce363b" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53cf4a6397aa43afafa769c9e92f9618" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38131 } health_report { overall_health: UNKNOWN } } }
W20260810 23:32:25.134047 17658 tablet.cc:2369] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:32:25.144079 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:25.142193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.9.48.193:45579)
0810 23:32:25.142364 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:25.142369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:25.142388 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:32:25.142480 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:25.142484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:25.142564 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:25.142671 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:25.143544 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:25.143548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:25.143552 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:25.143568 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:25.143600 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:25.143612 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:25.143717 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:25.143812 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 23:32:25.143861 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:32:25.152402 17581 raft_consensus.cc:1261] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618 [term 1 FOLLOWER]: Refusing update from remote peer da031898cdc242a98d5db97a361de40c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:25.152397 17781 raft_consensus.cc:1261] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b [term 1 FOLLOWER]: Refusing update from remote peer da031898cdc242a98d5db97a361de40c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:25.153395 17891 consensus_queue.cc:1035] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [LEADER]: Connected to new peer: Peer: permanent_uuid: "53cf4a6397aa43afafa769c9e92f9618" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:25.153710 17897 consensus_queue.cc:1035] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e50b47491d3240818e301fbfc3ce363b" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:25.165711 17901 mvcc.cc:205] Tried to move back new op lower bound from 7317113836133961728 to 7317113835928547328. Current Snapshot: MvccSnapshot[applied={T|T < 7317113836133961728}]
I20260810 23:32:25.181221 17151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:25.181537 17151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:25.185213 17151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aebe8e4e14314e5f8b90dd9f1cf40fd3: Sending DeleteTablet for 3 replicas of tablet 8500518e0c924e6992b4e8d42cdeea9b
I20260810 23:32:25.186393 17344 tablet_service.cc:1514] Processing DeleteTablet for tablet 8500518e0c924e6992b4e8d42cdeea9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:25 UTC) from {username='slave'} at 127.0.0.1:43298
I20260810 23:32:25.186827 17555 tablet_service.cc:1514] Processing DeleteTablet for tablet 8500518e0c924e6992b4e8d42cdeea9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:25 UTC) from {username='slave'} at 127.0.0.1:58120
I20260810 23:32:25.187067 17750 tablet_service.cc:1514] Processing DeleteTablet for tablet 8500518e0c924e6992b4e8d42cdeea9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:25 UTC) from {username='slave'} at 127.0.0.1:36784
I20260810 23:32:25.189585 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 17242
I20260810 23:32:25.197168 17917 tablet_replica.cc:330] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618: stopping tablet replica
I20260810 23:32:25.197356 17915 tablet_replica.cc:330] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c: stopping tablet replica
I20260810 23:32:25.197731 17915 raft_consensus.cc:2229] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:25.197760 17917 raft_consensus.cc:2229] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:25.198290 17917 raft_consensus.cc:2258] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:25.198310 17915 raft_consensus.cc:2258] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:25.200903 17917 ts_tablet_manager.cc:1905] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:25.201555 17918 tablet_replica.cc:330] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b: stopping tablet replica
I20260810 23:32:25.201577 17915 ts_tablet_manager.cc:1905] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:25.201958 17918 raft_consensus.cc:2229] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:25.202461 17918 raft_consensus.cc:2258] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:25.204674 17918 ts_tablet_manager.cc:1905] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:25.209157 17917 ts_tablet_manager.cc:1918] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:25.209388 17917 log.cc:1196] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/8500518e0c924e6992b4e8d42cdeea9b
I20260810 23:32:25.209968 17917 ts_tablet_manager.cc:1939] T 8500518e0c924e6992b4e8d42cdeea9b P 53cf4a6397aa43afafa769c9e92f9618: Deleting consensus metadata
I20260810 23:32:25.211596 17918 ts_tablet_manager.cc:1918] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:25.211774 17918 log.cc:1196] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/8500518e0c924e6992b4e8d42cdeea9b
I20260810 23:32:25.211958 17122 catalog_manager.cc:4841] TS 53cf4a6397aa43afafa769c9e92f9618 (127.9.48.194:38131): tablet 8500518e0c924e6992b4e8d42cdeea9b (table test-table [id=326b801c46514a3cbd100804caba9e13]) successfully deleted
I20260810 23:32:25.212317 17918 ts_tablet_manager.cc:1939] T 8500518e0c924e6992b4e8d42cdeea9b P e50b47491d3240818e301fbfc3ce363b: Deleting consensus metadata
I20260810 23:32:25.212847 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:25.192002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.9.48.193:45579)
0810 23:32:25.192224 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:25.192234 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:25.206500 (+ 14266us) server_negotiation.cc:407] Connection header received
0810 23:32:25.206977 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:25.206985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:25.207059 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:25.207131 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:25.211209 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:25.211215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:25.211220 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:25.211240 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:25.211277 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:25.211292 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:25.211389 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:25.212397 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0810 23:32:25.212464 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:32:25.213529 17442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:25.214094 17123 catalog_manager.cc:4841] TS e50b47491d3240818e301fbfc3ce363b (127.9.48.195:45705): tablet 8500518e0c924e6992b4e8d42cdeea9b (table test-table [id=326b801c46514a3cbd100804caba9e13]) successfully deleted
I20260810 23:32:25.311337 17915 ts_tablet_manager.cc:1918] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:25.311672 17915 log.cc:1196] T 8500518e0c924e6992b4e8d42cdeea9b P da031898cdc242a98d5db97a361de40c: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/8500518e0c924e6992b4e8d42cdeea9b
I20260810 23:32:25.312096 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 17242
W20260810 23:32:25.342667 17122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.48.193:45579 (error 108)
I20260810 23:32:25.343021 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 17457
I20260810 23:32:25.345129 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:25.344352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:45579 (local address 127.0.0.1:43328)
0810 23:32:25.344658 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:25.344914 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 23:32:25.345381 17122 catalog_manager.cc:4584] TS da031898cdc242a98d5db97a361de40c (127.9.48.193:45579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8500518e0c924e6992b4e8d42cdeea9b: Network error: Client connection negotiation failed: client connection to 127.9.48.193:45579: connect: Connection refused (error 111)
I20260810 23:32:25.346577 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:25.343884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58134 (local address 127.9.48.194:38131)
0810 23:32:25.344060 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:25.344086 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:25.344116 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:32:25.344408 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:25.344415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:25.344581 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:25.344757 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:25.345797 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:25.345803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:25.345807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:25.345826 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:25.345865 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:25.345877 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:25.346019 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:25.346201 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 23:32:25.346267 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:32:25.347496 17642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:25.400629 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:25.400105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:45579 (local address 127.0.0.1:36328)
0810 23:32:25.400292 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:25.400386 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 23:32:25.401731 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:25.401375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:45579 (local address 127.0.0.1:36330)
0810 23:32:25.401533 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:25.401625 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 23:32:25.437131 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 17457
I20260810 23:32:25.462644 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:25.462161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:45579 (local address 127.0.0.1:36334)
0810 23:32:25.462381 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:25.462488 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 23:32:25.463886 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:25.463528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:45579 (local address 127.0.0.1:36338)
0810 23:32:25.463672 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:25.463743 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 23:32:25.471221 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 17660
I20260810 23:32:25.474634 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:25.471985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46140 (local address 127.9.48.195:45705)
0810 23:32:25.472270 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:25.472279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:25.472306 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:32:25.472554 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:25.472565 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:25.472707 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:25.472850 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:25.473857 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:25.473865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:25.473870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:25.473901 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:25.473939 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:25.473954 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:25.474162 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:25.474226 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 23:32:25.474297 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260810 23:32:25.475308 17834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:25.544281 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:25.543777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:45579 (local address 127.0.0.1:36340)
0810 23:32:25.544013 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:25.544118 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260810 23:32:25.545500 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:25.545079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:45579 (local address 127.0.0.1:36350)
0810 23:32:25.545301 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:25.545370 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:45579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260810 23:32:25.563874 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 17660
I20260810 23:32:25.592336 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 17104
I20260810 23:32:25.593604 17202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:25.699481 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 17104
2026-08-10T23:32:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2841 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 23:32:25.755201 9411 mini_oidc.cc:152] Starting JWKS server
I20260810 23:32:25.758400 9411 webserver.cc:460] Webserver started at https://127.0.0.1:46311/ using document root <none> and password file <none>
I20260810 23:32:25.758811 9411 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:32:25.759541 9411 webserver.cc:460] Webserver started at http://127.0.0.1:36081/ using document root <none> and password file <none>
2026-08-10T23:32:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:25Z Disabled control of system clock
I20260810 23:32:25.794122 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:45551
--webserver_interface=127.9.48.254
--webserver_port=0
--builtin_ntp_servers=127.9.48.212:39913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:45551
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46311/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/testchainca.cert with env {}
W20260810 23:32:26.126443 17932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:26.126807 17932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:26.126986 17932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:26.134895 17932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:26.135046 17932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:26.135146 17932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:26.135251 17932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:26.135336 17932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:32:26.135428 17932 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:32:26.135506 17932 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46311/jwks.json
I20260810 23:32:26.145399 17932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:39913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:45551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:45551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46311/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:26.146733 17932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:26.148072 17932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:26.160142 17939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:26.160084 17941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:26.160192 17943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:26.161039 17932 server_base.cc:1034] running on GCE node
I20260810 23:32:26.161805 17932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:26.163180 17932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:26.164390 17932 hybrid_clock.cc:648] HybridClock initialized: now 1786404746164344 us; error 40 us; skew 500 ppm
I20260810 23:32:26.164770 17932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:32:26.167057 17932 webserver.cc:460] Webserver started at http://127.9.48.254:45793/ using document root <none> and password file <none>
I20260810 23:32:26.167706 17932 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:26.167794 17932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:26.168063 17932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:26.171025 17932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "8d377f46a52f479a91696e70fc4a71c7"
format_stamp: "Formatted at 2026-08-10 23:32:26 on dist-test-slave-04bb"
server_key: "186d34f5c9b867a502fee721c53c61ff"
server_key_iv: "ad1a2b5bf810bee62fe117d863279ac9"
server_key_version: "encryptionkey@0"
I20260810 23:32:26.171763 17932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "8d377f46a52f479a91696e70fc4a71c7"
format_stamp: "Formatted at 2026-08-10 23:32:26 on dist-test-slave-04bb"
server_key: "186d34f5c9b867a502fee721c53c61ff"
server_key_iv: "ad1a2b5bf810bee62fe117d863279ac9"
server_key_version: "encryptionkey@0"
I20260810 23:32:26.176883 17932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:32:26.180313 17955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:26.181771 17932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:26.181922 17932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "8d377f46a52f479a91696e70fc4a71c7"
format_stamp: "Formatted at 2026-08-10 23:32:26 on dist-test-slave-04bb"
server_key: "186d34f5c9b867a502fee721c53c61ff"
server_key_iv: "ad1a2b5bf810bee62fe117d863279ac9"
server_key_version: "encryptionkey@0"
I20260810 23:32:26.182101 17932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:26.205063 17932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:26.205849 17932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:26.206123 17932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:26.225144 17932 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:45551
I20260810 23:32:26.225165 18024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:45551 every 8 connection(s)
I20260810 23:32:26.226485 17932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:26.230314 18026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:26.234007 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 17932
I20260810 23:32:26.234225 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:32:26.234894 9411 external_mini_cluster.cc:1428] Setting key 32471edfe3924d8f28d4cd0bef164bd5
I20260810 23:32:26.244470 18026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7: Bootstrap starting.
I20260810 23:32:26.248479 18026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:26.249866 18026 log.cc:824] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:26.250078 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:26.236880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.9.48.254:45551)
0810 23:32:26.237964 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:26.237978 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:26.238042 (+ 64us) server_negotiation.cc:407] Connection header received
0810 23:32:26.239782 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:26.239816 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:26.240448 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:26.240904 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:26.241504 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:26.243441 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:26.244858 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:26.245847 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:26.246031 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:26.247174 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:26.247184 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:26.247202 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:26.247949 (+ 747us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:26.248067 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:26.248095 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:26.248219 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:26.248406 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 23:32:26.248759 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":311,"threads_started":1}
I20260810 23:32:26.253397 18026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7: No bootstrap required, opened a new log
I20260810 23:32:26.254393 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:32:26.261484 18026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d377f46a52f479a91696e70fc4a71c7" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45551 } }
I20260810 23:32:26.261830 18026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:26.261951 18026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d377f46a52f479a91696e70fc4a71c7, State: Initialized, Role: FOLLOWER
I20260810 23:32:26.262583 18026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d377f46a52f479a91696e70fc4a71c7" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45551 } }
I20260810 23:32:26.262789 18026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:32:26.262887 18026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:32:26.263016 18026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:26.265578 18026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d377f46a52f479a91696e70fc4a71c7" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45551 } }
I20260810 23:32:26.266036 18026 leader_election.cc:302] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d377f46a52f479a91696e70fc4a71c7; no voters:
I20260810 23:32:26.266551 18026 leader_election.cc:288] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:32:26.266701 18033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:26.267427 18033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [term 1 LEADER]: Becoming Leader. State: Replica: 8d377f46a52f479a91696e70fc4a71c7, State: Running, Role: LEADER
I20260810 23:32:26.267994 18033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d377f46a52f479a91696e70fc4a71c7" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45551 } }
I20260810 23:32:26.268829 18026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:26.271239 18034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d377f46a52f479a91696e70fc4a71c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d377f46a52f479a91696e70fc4a71c7" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45551 } } }
I20260810 23:32:26.271235 18036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d377f46a52f479a91696e70fc4a71c7. Latest consensus state: current_term: 1 leader_uuid: "8d377f46a52f479a91696e70fc4a71c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d377f46a52f479a91696e70fc4a71c7" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45551 } } }
I20260810 23:32:26.271657 18034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:26.271726 18036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:26.273236 18046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:26.277632 18046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:26.286480 18046 catalog_manager.cc:1347] Generated new cluster ID: 1888e80fcfbc453d9a63537db226f647
I20260810 23:32:26.286578 18046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:26.296145 18046 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:32:26.297405 18046 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:26.302474 18046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d377f46a52f479a91696e70fc4a71c7: Generated new TSK 0
I20260810 23:32:26.303340 18046 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 23:32:26.347064 18070 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 23:32:26.347276 18070 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 23:32:26.350128 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:26.326188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.9.48.254:45551)
0810 23:32:26.326392 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:26.326399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:26.326998 (+ 599us) server_negotiation.cc:407] Connection header received
0810 23:32:26.327330 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:26.327335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:26.327437 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:26.327596 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:32:26.328161 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:26.329330 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:26.330755 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:26.331459 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:26.331599 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:26.332071 (+ 472us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:32:26.349448 (+ 17377us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 23:32:26.349799 (+ 351us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60712: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 23:32:26.350261 18030 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60712: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 23:32:26.352033 18069 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:26.326077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:45551 (local address 127.0.0.1:60712)
0810 23:32:26.326867 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:26.326883 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:26.327052 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:26.327615 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:26.327619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:26.327628 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:32:26.327972 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:26.327981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:26.329473 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:26.329477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:26.330614 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:26.330624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:26.331615 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:26.331619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:26.331741 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:26.331795 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:32:26.349745 (+ 17950us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 23:32:26.350019 (+ 274us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.48.254:45551: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":419,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.48.254:45551: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46311/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 23:32:26.354337 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 17932
I20260810 23:32:26.355460 18016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:26.453222 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 17932
2026-08-10T23:32:26Z chronyd exiting
I20260810 23:32:26.512878 9411 test_util.cc:175] -----------------------------------------------
I20260810 23:32:26.513041 9411 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786404698557218-9411-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (763 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 23:32:26.517848 9411 mini_oidc.cc:152] Starting JWKS server
I20260810 23:32:26.519718 9411 webserver.cc:460] Webserver started at https://127.0.0.1:33131/ using document root <none> and password file <none>
I20260810 23:32:26.519986 9411 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:32:26.520704 9411 webserver.cc:460] Webserver started at http://127.0.0.1:36157/ using document root <none> and password file <none>
2026-08-10T23:32:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:26Z Disabled control of system clock
I20260810 23:32:26.555310 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:38229
--webserver_interface=127.9.48.254
--webserver_port=0
--builtin_ntp_servers=127.9.48.212:41749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:38229
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33131/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/testchainca.cert with env {}
W20260810 23:32:26.892791 18087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:26.893170 18087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:26.893370 18087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:26.900815 18087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:26.900974 18087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:26.901070 18087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:26.901180 18087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:26.901268 18087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:32:26.901404 18087 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:32:26.901484 18087 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33131/jwks.json
I20260810 23:32:26.911716 18087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:41749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:38229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:38229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33131/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:26.913201 18087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:26.914644 18087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:26.927403 18098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:26.927455 18099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:26.927361 18102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:26.929168 18087 server_base.cc:1034] running on GCE node
I20260810 23:32:26.929934 18087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:26.931375 18087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:26.932624 18087 hybrid_clock.cc:648] HybridClock initialized: now 1786404746932574 us; error 48 us; skew 500 ppm
I20260810 23:32:26.932981 18087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:32:26.935401 18087 webserver.cc:460] Webserver started at http://127.9.48.254:44793/ using document root <none> and password file <none>
I20260810 23:32:26.936115 18087 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:26.936255 18087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:26.936666 18087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:26.939570 18087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "d47f322bd3994f04bbd97a36fa2353a9"
format_stamp: "Formatted at 2026-08-10 23:32:26 on dist-test-slave-04bb"
server_key: "517c098443a8dd1961ace309e581ff16"
server_key_iv: "69d0daf26049f75bd466b3c37455e4e5"
server_key_version: "encryptionkey@0"
I20260810 23:32:26.940387 18087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "d47f322bd3994f04bbd97a36fa2353a9"
format_stamp: "Formatted at 2026-08-10 23:32:26 on dist-test-slave-04bb"
server_key: "517c098443a8dd1961ace309e581ff16"
server_key_iv: "69d0daf26049f75bd466b3c37455e4e5"
server_key_version: "encryptionkey@0"
I20260810 23:32:26.945600 18087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260810 23:32:26.949136 18112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:26.950572 18087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:32:26.950757 18087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "d47f322bd3994f04bbd97a36fa2353a9"
format_stamp: "Formatted at 2026-08-10 23:32:26 on dist-test-slave-04bb"
server_key: "517c098443a8dd1961ace309e581ff16"
server_key_iv: "69d0daf26049f75bd466b3c37455e4e5"
server_key_version: "encryptionkey@0"
I20260810 23:32:26.950986 18087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:26.966038 18087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:26.966828 18087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:26.967144 18087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:26.986555 18087 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:38229
I20260810 23:32:26.986593 18188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:38229 every 8 connection(s)
I20260810 23:32:26.987938 18087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:26.992071 18189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:26.995751 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 18087
I20260810 23:32:26.995973 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:32:26.996932 9411 external_mini_cluster.cc:1428] Setting key 7b5623ae6982f7334b86c923cfabd53c
I20260810 23:32:27.005577 18189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9: Bootstrap starting.
I20260810 23:32:27.008948 18189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:27.010231 18189 log.cc:824] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:27.010555 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:26.999136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.9.48.254:38229)
0810 23:32:27.000132 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:27.000143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:27.000183 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:32:27.001951 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:27.001975 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:27.002453 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:27.002865 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:27.003452 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:27.004762 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:27.006106 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:27.006774 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:27.006981 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:27.008190 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:27.008197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:27.008210 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:27.008804 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:27.008913 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:27.008937 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:27.009068 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:27.009301 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 23:32:27.009610 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":355,"threads_started":1}
I20260810 23:32:27.013996 18189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9: No bootstrap required, opened a new log
I20260810 23:32:27.014991 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:32:27.023293 18189 raft_consensus.cc:348] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f322bd3994f04bbd97a36fa2353a9" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38229 } }
I20260810 23:32:27.023682 18189 raft_consensus.cc:374] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:27.023803 18189 raft_consensus.cc:729] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d47f322bd3994f04bbd97a36fa2353a9, State: Initialized, Role: FOLLOWER
I20260810 23:32:27.024413 18189 consensus_queue.cc:260] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f322bd3994f04bbd97a36fa2353a9" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38229 } }
I20260810 23:32:27.024751 18189 raft_consensus.cc:388] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:32:27.024920 18189 raft_consensus.cc:482] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:32:27.025157 18189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:27.027644 18189 raft_consensus.cc:504] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f322bd3994f04bbd97a36fa2353a9" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38229 } }
I20260810 23:32:27.028223 18189 leader_election.cc:302] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d47f322bd3994f04bbd97a36fa2353a9; no voters:
I20260810 23:32:27.028878 18189 leader_election.cc:288] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:32:27.029021 18195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:27.029846 18195 raft_consensus.cc:686] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [term 1 LEADER]: Becoming Leader. State: Replica: d47f322bd3994f04bbd97a36fa2353a9, State: Running, Role: LEADER
I20260810 23:32:27.030444 18195 consensus_queue.cc:237] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f322bd3994f04bbd97a36fa2353a9" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38229 } }
I20260810 23:32:27.031113 18189 sys_catalog.cc:564] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:27.033272 18198 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d47f322bd3994f04bbd97a36fa2353a9. Latest consensus state: current_term: 1 leader_uuid: "d47f322bd3994f04bbd97a36fa2353a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f322bd3994f04bbd97a36fa2353a9" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38229 } } }
I20260810 23:32:27.033983 18198 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:27.034467 18197 sys_catalog.cc:455] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d47f322bd3994f04bbd97a36fa2353a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d47f322bd3994f04bbd97a36fa2353a9" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38229 } } }
I20260810 23:32:27.035656 18197 sys_catalog.cc:458] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:27.042002 18218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:27.047214 18218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:27.055989 18218 catalog_manager.cc:1347] Generated new cluster ID: 7db374805e0845a289c8b0869fa4ba89
I20260810 23:32:27.056116 18218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:27.070403 18218 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:32:27.071650 18218 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:27.082826 18218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d47f322bd3994f04bbd97a36fa2353a9: Generated new TSK 0
I20260810 23:32:27.083835 18218 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 23:32:27.118253 18238 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 23:32:27.118444 18238 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 23:32:27.119397 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:27.098393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.9.48.254:38229)
0810 23:32:27.098660 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:27.098668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:27.099102 (+ 434us) server_negotiation.cc:407] Connection header received
0810 23:32:27.099388 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:27.099393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:27.099458 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:27.099583 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:32:27.100168 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:27.101103 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:27.103211 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:27.103731 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:27.103837 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:27.104455 (+ 618us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:32:27.118874 (+ 14419us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 23:32:27.119172 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57848: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 23:32:27.119542 18193 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57848: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 23:32:27.119813 18237 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:27.098273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:38229 (local address 127.0.0.1:57848)
0810 23:32:27.098971 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:27.099005 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:27.099166 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:27.099639 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:27.099645 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:27.099658 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:32:27.099978 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:27.099991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:27.101267 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:27.101273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:27.103036 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:27.103052 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:27.103943 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:27.103951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:27.104151 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:27.104200 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:32:27.119096 (+ 14896us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 23:32:27.119377 (+ 281us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.48.254:38229: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":366,"threads_started":1}
I20260810 23:32:27.120844 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 18087
I20260810 23:32:27.122080 18181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:27.216234 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 18087
2026-08-10T23:32:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (766 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 23:32:27.283360 9411 mini_oidc.cc:152] Starting JWKS server
I20260810 23:32:27.285413 9411 webserver.cc:460] Webserver started at https://127.0.0.1:35659/ using document root <none> and password file <none>
I20260810 23:32:27.285686 9411 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:32:27.286335 9411 webserver.cc:460] Webserver started at http://127.0.0.1:40137/ using document root <none> and password file <none>
2026-08-10T23:32:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:27Z Disabled control of system clock
I20260810 23:32:27.320955 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:38741
--webserver_interface=127.9.48.254
--webserver_port=0
--builtin_ntp_servers=127.9.48.212:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:38741
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35659/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 23:32:27.663635 18247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:27.663993 18247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:27.664153 18247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:27.671242 18247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:27.671391 18247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:27.671486 18247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:27.671597 18247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:27.671685 18247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:32:27.671777 18247 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:32:27.671839 18247 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35659/jwks.json
I20260810 23:32:27.680639 18247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:38741
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:38741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35659/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:27.681910 18247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:27.683722 18247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:27.695749 18255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:27.695783 18256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:27.695817 18260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:27.696936 18247 server_base.cc:1034] running on GCE node
I20260810 23:32:27.697697 18247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:27.699071 18247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:27.700318 18247 hybrid_clock.cc:648] HybridClock initialized: now 1786404747700285 us; error 43 us; skew 500 ppm
I20260810 23:32:27.700731 18247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:32:27.703065 18247 webserver.cc:460] Webserver started at http://127.9.48.254:42039/ using document root <none> and password file <none>
I20260810 23:32:27.703756 18247 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:27.703894 18247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:27.704232 18247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:27.707111 18247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "03687ebaf481496798447396c0b93239"
format_stamp: "Formatted at 2026-08-10 23:32:27 on dist-test-slave-04bb"
server_key: "d2d098e568008455388fc9d4dc675371"
server_key_iv: "d7592117a2eff1d299208cd86cba52d9"
server_key_version: "encryptionkey@0"
I20260810 23:32:27.707949 18247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "03687ebaf481496798447396c0b93239"
format_stamp: "Formatted at 2026-08-10 23:32:27 on dist-test-slave-04bb"
server_key: "d2d098e568008455388fc9d4dc675371"
server_key_iv: "d7592117a2eff1d299208cd86cba52d9"
server_key_version: "encryptionkey@0"
I20260810 23:32:27.713019 18247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:32:27.716320 18266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:27.717636 18247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:27.717832 18247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "03687ebaf481496798447396c0b93239"
format_stamp: "Formatted at 2026-08-10 23:32:27 on dist-test-slave-04bb"
server_key: "d2d098e568008455388fc9d4dc675371"
server_key_iv: "d7592117a2eff1d299208cd86cba52d9"
server_key_version: "encryptionkey@0"
I20260810 23:32:27.718053 18247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:27.732072 18247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:27.732944 18247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:27.733261 18247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:27.751986 18247 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:38741
I20260810 23:32:27.752007 18342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:38741 every 8 connection(s)
I20260810 23:32:27.753435 18247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:27.757107 18343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:27.760211 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 18247
I20260810 23:32:27.760471 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:32:27.761361 9411 external_mini_cluster.cc:1428] Setting key f8fab2cf422aae7f12a5e3fef64d795b
I20260810 23:32:27.769711 18343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239: Bootstrap starting.
I20260810 23:32:27.772959 18343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:27.774135 18343 log.cc:824] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:27.775328 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:27.763521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.9.48.254:38741)
0810 23:32:27.764504 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:27.764515 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:27.764609 (+ 94us) server_negotiation.cc:407] Connection header received
0810 23:32:27.766281 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:27.766311 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:27.766777 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:27.767189 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:27.767791 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:27.769127 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:27.770505 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:27.771208 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:27.771431 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:27.772652 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:27.772662 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:27.772676 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:27.773268 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:27.773377 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:27.773399 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:27.773535 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:27.773759 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 23:32:27.774084 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":355,"threads_started":1}
I20260810 23:32:27.777251 18343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239: No bootstrap required, opened a new log
I20260810 23:32:27.779106 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:32:27.786057 18343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03687ebaf481496798447396c0b93239" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38741 } }
I20260810 23:32:27.786566 18343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:27.786715 18343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03687ebaf481496798447396c0b93239, State: Initialized, Role: FOLLOWER
I20260810 23:32:27.787422 18343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03687ebaf481496798447396c0b93239" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38741 } }
I20260810 23:32:27.787719 18343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:32:27.787907 18343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:32:27.788120 18343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:27.790637 18343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03687ebaf481496798447396c0b93239" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38741 } }
I20260810 23:32:27.791414 18343 leader_election.cc:302] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03687ebaf481496798447396c0b93239; no voters:
I20260810 23:32:27.792117 18343 leader_election.cc:288] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:32:27.792253 18353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:27.793080 18353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [term 1 LEADER]: Becoming Leader. State: Replica: 03687ebaf481496798447396c0b93239, State: Running, Role: LEADER
I20260810 23:32:27.793649 18353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03687ebaf481496798447396c0b93239" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38741 } }
I20260810 23:32:27.794556 18343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:27.796898 18356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03687ebaf481496798447396c0b93239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03687ebaf481496798447396c0b93239" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38741 } } }
I20260810 23:32:27.797310 18356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:27.797525 18357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03687ebaf481496798447396c0b93239. Latest consensus state: current_term: 1 leader_uuid: "03687ebaf481496798447396c0b93239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03687ebaf481496798447396c0b93239" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38741 } } }
I20260810 23:32:27.797897 18357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:27.799782 18367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:27.805895 18367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:27.814980 18367 catalog_manager.cc:1347] Generated new cluster ID: 290bb8e85be84d63a9ce300716cb4526
I20260810 23:32:27.815167 18367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:27.837009 18367 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:32:27.838790 18367 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:27.861181 18367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03687ebaf481496798447396c0b93239: Generated new TSK 0
I20260810 23:32:27.862278 18367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:32:27.887518 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:27.878690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.9.48.254:38741)
0810 23:32:27.878877 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:27.878884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:27.879345 (+ 461us) server_negotiation.cc:407] Connection header received
0810 23:32:27.879619 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:27.879625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:27.879740 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:27.879905 (+ 165us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:32:27.880504 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:27.881713 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:27.883088 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:27.883756 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:27.883882 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:27.884382 (+ 500us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:32:27.886837 (+ 2455us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35659/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 23:32:27.887189 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37850: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35659/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 23:32:27.887676 18350 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37850: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35659/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 23:32:27.888715 18390 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:27.878529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:38741 (local address 127.0.0.1:37850)
0810 23:32:27.879218 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:27.879243 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:27.879419 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:27.879993 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:27.880004 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:27.880024 (+ 20us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:32:27.880344 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:27.880353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:27.881874 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:27.881878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:27.882938 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:27.882950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:27.883950 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:27.883955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:27.884084 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:27.884117 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:32:27.887090 (+ 2973us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35659/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 23:32:27.887431 (+ 341us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.48.254:38741: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35659/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":1030,"threads_started":1}
I20260810 23:32:27.891278 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 18247
I20260810 23:32:27.893003 18338 generic_service.cc:196] Checking for leaks (request via RPC)
E20260810 23:32:27.902544 18391 webserver.cc:533] Webserver: sslize failed:
E20260810 23:32:27.902659 18391 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 23:32:27.985922 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 18247
2026-08-10T23:32:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (766 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T23:32:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:28Z Disabled control of system clock
I20260810 23:32:28.086107 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:41185
--webserver_interface=127.9.48.254
--webserver_port=0
--builtin_ntp_servers=127.9.48.212:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:41185
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/testchainca.cert with env {}
W20260810 23:32:28.421841 18402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:28.422137 18402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:28.422233 18402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:28.429358 18402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:28.429461 18402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:28.429502 18402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:28.429562 18402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:28.429600 18402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:32:28.429663 18402 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:32:28.429704 18402 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 23:32:28.439544 18402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:41185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:41185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:28.440896 18402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:28.442149 18402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:28.453943 18409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:28.454010 18410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:28.454041 18415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:28.454851 18402 server_base.cc:1034] running on GCE node
I20260810 23:32:28.455626 18402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:28.457105 18402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:28.458335 18402 hybrid_clock.cc:648] HybridClock initialized: now 1786404748458303 us; error 61 us; skew 500 ppm
I20260810 23:32:28.458698 18402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:32:28.461069 18402 webserver.cc:460] Webserver started at http://127.9.48.254:43613/ using document root <none> and password file <none>
I20260810 23:32:28.461759 18402 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:28.461895 18402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:28.462222 18402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:28.465157 18402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "2173a513ae6340a887f61080ee62828b"
format_stamp: "Formatted at 2026-08-10 23:32:28 on dist-test-slave-04bb"
server_key: "f6abf8c72282818f86a692b001f0a412"
server_key_iv: "0fb0856d753c9489ea84aa29518efe3b"
server_key_version: "encryptionkey@0"
I20260810 23:32:28.466002 18402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "2173a513ae6340a887f61080ee62828b"
format_stamp: "Formatted at 2026-08-10 23:32:28 on dist-test-slave-04bb"
server_key: "f6abf8c72282818f86a692b001f0a412"
server_key_iv: "0fb0856d753c9489ea84aa29518efe3b"
server_key_version: "encryptionkey@0"
I20260810 23:32:28.470991 18402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:32:28.474362 18423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:28.475633 18402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:32:28.475854 18402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "2173a513ae6340a887f61080ee62828b"
format_stamp: "Formatted at 2026-08-10 23:32:28 on dist-test-slave-04bb"
server_key: "f6abf8c72282818f86a692b001f0a412"
server_key_iv: "0fb0856d753c9489ea84aa29518efe3b"
server_key_version: "encryptionkey@0"
I20260810 23:32:28.476071 18402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:28.498448 18402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:28.499343 18402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:28.499652 18402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:28.517807 18402 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:41185
I20260810 23:32:28.517834 18490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:41185 every 8 connection(s)
I20260810 23:32:28.519155 18402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:28.522359 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 18402
I20260810 23:32:28.522571 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:32:28.523059 18492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:28.523337 9411 external_mini_cluster.cc:1428] Setting key dc81d2ed08a8aba5ac8cb89a2bda8e38
I20260810 23:32:28.538520 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:28.525321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56110 (local address 127.9.48.254:41185)
0810 23:32:28.526451 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:28.526465 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:28.526517 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:32:28.528317 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:28.528346 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:28.528970 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:28.529406 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:28.529958 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:28.531996 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:28.533356 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:28.534386 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:28.534573 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:28.535711 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:28.535721 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:28.535739 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:32:28.536468 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:28.536582 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:32:28.536609 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:28.536711 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:28.536937 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 23:32:28.537240 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":365,"threads_started":1}
I20260810 23:32:28.538516 18492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b: Bootstrap starting.
I20260810 23:32:28.542408 18492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:28.542611 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:32:28.543481 18492 log.cc:824] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:28.546718 18492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b: No bootstrap required, opened a new log
I20260810 23:32:28.554680 18492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2173a513ae6340a887f61080ee62828b" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41185 } }
I20260810 23:32:28.555027 18492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:28.555127 18492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2173a513ae6340a887f61080ee62828b, State: Initialized, Role: FOLLOWER
I20260810 23:32:28.555680 18492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2173a513ae6340a887f61080ee62828b" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41185 } }
I20260810 23:32:28.555967 18492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:32:28.556120 18492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:32:28.556293 18492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:28.558828 18492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2173a513ae6340a887f61080ee62828b" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41185 } }
I20260810 23:32:28.559330 18492 leader_election.cc:302] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2173a513ae6340a887f61080ee62828b; no voters:
I20260810 23:32:28.559944 18492 leader_election.cc:288] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:32:28.560106 18500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:28.560855 18500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [term 1 LEADER]: Becoming Leader. State: Replica: 2173a513ae6340a887f61080ee62828b, State: Running, Role: LEADER
I20260810 23:32:28.561442 18500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2173a513ae6340a887f61080ee62828b" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41185 } }
I20260810 23:32:28.562098 18492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:28.564651 18503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2173a513ae6340a887f61080ee62828b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2173a513ae6340a887f61080ee62828b" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41185 } } }
I20260810 23:32:28.564570 18504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2173a513ae6340a887f61080ee62828b. Latest consensus state: current_term: 1 leader_uuid: "2173a513ae6340a887f61080ee62828b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2173a513ae6340a887f61080ee62828b" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41185 } } }
I20260810 23:32:28.565042 18503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:28.565099 18504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:28.567243 18513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:28.572337 18513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:28.580989 18513 catalog_manager.cc:1347] Generated new cluster ID: 751a64d3368a493b9985fa9942c8a4cd
I20260810 23:32:28.581131 18513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:28.601457 18513 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:32:28.603274 18513 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:28.615770 18513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2173a513ae6340a887f61080ee62828b: Generated new TSK 0
I20260810 23:32:28.616641 18513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:32:29.105881 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:28.630845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56118 (local address 127.9.48.254:41185)
0810 23:32:28.631074 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:28.631084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:28.631567 (+ 483us) server_negotiation.cc:407] Connection header received
0810 23:32:28.631787 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:28.631795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:28.631863 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:28.631987 (+ 124us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:32:28.632610 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:28.633325 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:28.635453 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:28.635906 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:28.635999 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:28.636612 (+ 613us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:32:29.105303 (+468691us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 23:32:29.105648 (+ 345us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56118: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 23:32:29.106035 18497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56118: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 23:32:29.106256 18540 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:28.630811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:41185 (local address 127.0.0.1:56118)
0810 23:32:28.631453 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:28.631473 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:28.631625 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:28.632033 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:28.632039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:28.632053 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:32:28.632395 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:28.632409 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:28.633477 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:28.633483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:28.635299 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:28.635315 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:28.636076 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:28.636086 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:28.636270 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:28.636314 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:32:29.105607 (+469293us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 23:32:29.105893 (+ 286us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.48.254:41185: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":330,"threads_started":1}
I20260810 23:32:29.108067 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 18402
I20260810 23:32:29.109246 18483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:29.213389 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 18402
2026-08-10T23:32:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1209 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (424 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (427 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:32:30 dist-test-slave-04bb krb5kdc[18564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:32:30 dist-test-slave-04bb krb5kdc[18564](info): set up 2 sockets
Aug 10 23:32:30 dist-test-slave-04bb krb5kdc[18564](info): commencing operation
krb5kdc: starting...
W20260810 23:32:31.429363 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.317s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:32:31 dist-test-slave-04bb krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:32:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:32:31.845258 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:41791
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:41791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:32.483729 18582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:32.484246 18582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:32.484421 18582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:32.497805 18582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:32.497915 18582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:32.497961 18582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:32.498034 18582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:32.498118 18582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:32.516362 18582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:41791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:41791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:32.518880 18582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:32.521618 18582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:32.543300 18589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:32.543246 18588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:32.543233 18593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:32.545153 18582 server_base.cc:1034] running on GCE node
I20260810 23:32:32.546733 18582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:32.549535 18582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:32.550966 18582 hybrid_clock.cc:648] HybridClock initialized: now 1786404752550868 us; error 74 us; skew 500 ppm
I20260810 23:32:32.551579 18582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:32 dist-test-slave-04bb krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:32.562202 18582 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:32:32.565076 18582 webserver.cc:460] Webserver started at http://127.9.48.254:32845/ using document root <none> and password file <none>
I20260810 23:32:32.566596 18582 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:32.566883 18582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:32.567526 18582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:32.573441 18582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "4471b89a2fed4b5cb0e437bfb948451d"
format_stamp: "Formatted at 2026-08-10 23:32:32 on dist-test-slave-04bb"
server_key: "e79e925e9e6b7dcff7e14e4c4d66c3c7"
server_key_iv: "fd059b836d34f5253d4fddbc6d50e3a9"
server_key_version: "encryptionkey@0"
I20260810 23:32:32.575024 18582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "4471b89a2fed4b5cb0e437bfb948451d"
format_stamp: "Formatted at 2026-08-10 23:32:32 on dist-test-slave-04bb"
server_key: "e79e925e9e6b7dcff7e14e4c4d66c3c7"
server_key_iv: "fd059b836d34f5253d4fddbc6d50e3a9"
server_key_version: "encryptionkey@0"
I20260810 23:32:32.585584 18582 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.004s
I20260810 23:32:32.591956 18602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:32.594543 18582 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260810 23:32:32.594892 18582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "4471b89a2fed4b5cb0e437bfb948451d"
format_stamp: "Formatted at 2026-08-10 23:32:32 on dist-test-slave-04bb"
server_key: "e79e925e9e6b7dcff7e14e4c4d66c3c7"
server_key_iv: "fd059b836d34f5253d4fddbc6d50e3a9"
server_key_version: "encryptionkey@0"
I20260810 23:32:32.595273 18582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:32.624148 18582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:32.631724 18582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:32.632272 18582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:32.667652 18582 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:41791
I20260810 23:32:32.667680 18685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:41791 every 8 connection(s)
I20260810 23:32:32.670440 18582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:32.675117 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 18582
I20260810 23:32:32.675519 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:32:32.676885 9411 external_mini_cluster.cc:1428] Setting key cdb4b874b44157e5ddcb6466674ce9ed
I20260810 23:32:32.678995 18686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:32:32 dist-test-slave-04bb krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:32.707273 18686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d: Bootstrap starting.
I20260810 23:32:32.716488 18686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:32.718798 18686 log.cc:824] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:32.725524 18686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d: No bootstrap required, opened a new log
I20260810 23:32:32.726262 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:32.680482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.9.48.254:41791)
0810 23:32:32.682081 (+ 1599us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:32.682116 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:32.682177 (+ 61us) server_negotiation.cc:407] Connection header received
0810 23:32:32.685128 (+ 2951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:32.685211 (+ 83us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:32.686077 (+ 866us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:32.686660 (+ 583us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:32.690453 (+ 3793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:32.692873 (+ 2420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:32.696466 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:32.697847 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:32.698079 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:32.707601 (+ 9522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:32.707626 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:32.707664 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:32.707773 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:32.716860 (+ 9087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:32.718318 (+ 1458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:32.718365 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:32.718378 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:32.718833 (+ 455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:32.719603 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:32.719623 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:32.719652 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:32.720974 (+ 1322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:32.721378 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:32.722402 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0810 23:32:32.723215 (+ 813us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1108,"thread_start_us":666,"threads_started":1}
I20260810 23:32:32.741611 18686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4471b89a2fed4b5cb0e437bfb948451d" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41791 } }
I20260810 23:32:32.742281 18686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:32.742408 18686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4471b89a2fed4b5cb0e437bfb948451d, State: Initialized, Role: FOLLOWER
I20260810 23:32:32.743573 18686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4471b89a2fed4b5cb0e437bfb948451d" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41791 } }
I20260810 23:32:32.744138 18686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:32:32.744464 18686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:32:32.744989 18686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:32.749439 18686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4471b89a2fed4b5cb0e437bfb948451d" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41791 } }
I20260810 23:32:32.750507 18686 leader_election.cc:302] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4471b89a2fed4b5cb0e437bfb948451d; no voters:
I20260810 23:32:32.751734 18686 leader_election.cc:288] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:32:32.752055 18692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:32.753650 18692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [term 1 LEADER]: Becoming Leader. State: Replica: 4471b89a2fed4b5cb0e437bfb948451d, State: Running, Role: LEADER
I20260810 23:32:32.754659 18692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4471b89a2fed4b5cb0e437bfb948451d" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41791 } }
I20260810 23:32:32.755467 18686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:32.761405 18695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4471b89a2fed4b5cb0e437bfb948451d. Latest consensus state: current_term: 1 leader_uuid: "4471b89a2fed4b5cb0e437bfb948451d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4471b89a2fed4b5cb0e437bfb948451d" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41791 } } }
I20260810 23:32:32.762251 18695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:32.763721 18693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4471b89a2fed4b5cb0e437bfb948451d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4471b89a2fed4b5cb0e437bfb948451d" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41791 } } }
I20260810 23:32:32.764410 18693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:32.768393 18702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:32.778529 18702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:32.799587 18702 catalog_manager.cc:1347] Generated new cluster ID: 3a9c8a1c0252447e8b25af9c2cc02c92
I20260810 23:32:32.799803 18702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:32.820732 18702 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:32:32.823109 18702 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:32.836299 18702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4471b89a2fed4b5cb0e437bfb948451d: Generated new TSK 0
I20260810 23:32:32.838639 18702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:32:32.943275 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41791
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:33.383261 18720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:33.383526 18720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:33.383695 18720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:33.391100 18720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:33.391209 18720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:33.391391 18720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:32:33.400943 18720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:33.402329 18720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:33.403842 18720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:33.417245 18730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:33.417315 18732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:33.417292 18734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:33.418295 18720 server_base.cc:1034] running on GCE node
I20260810 23:32:33.418951 18720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:33.419865 18720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:33.421140 18720 hybrid_clock.cc:648] HybridClock initialized: now 1786404753421043 us; error 46 us; skew 500 ppm
I20260810 23:32:33.421460 18720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:33 dist-test-slave-04bb krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404753, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:33.424939 18720 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:32:33.426456 18720 webserver.cc:460] Webserver started at http://127.9.48.193:45719/ using document root <none> and password file <none>
I20260810 23:32:33.427209 18720 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:33.427363 18720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:33.427740 18720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:33.430979 18720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "e9c55c2bf2dd4cd6b54084816fd65e7f"
format_stamp: "Formatted at 2026-08-10 23:32:33 on dist-test-slave-04bb"
server_key: "092607763387e75e8c77077bfc7aa769"
server_key_iv: "d626e9565e0a84648069c87daf1dcb19"
server_key_version: "encryptionkey@0"
I20260810 23:32:33.431890 18720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "e9c55c2bf2dd4cd6b54084816fd65e7f"
format_stamp: "Formatted at 2026-08-10 23:32:33 on dist-test-slave-04bb"
server_key: "092607763387e75e8c77077bfc7aa769"
server_key_iv: "d626e9565e0a84648069c87daf1dcb19"
server_key_version: "encryptionkey@0"
I20260810 23:32:33.437551 18720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 23:32:33.441155 18744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:33.442560 18720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:32:33.442762 18720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "e9c55c2bf2dd4cd6b54084816fd65e7f"
format_stamp: "Formatted at 2026-08-10 23:32:33 on dist-test-slave-04bb"
server_key: "092607763387e75e8c77077bfc7aa769"
server_key_iv: "d626e9565e0a84648069c87daf1dcb19"
server_key_version: "encryptionkey@0"
I20260810 23:32:33.442975 18720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:33.459532 18720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:33.463580 18720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:33.463896 18720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:33.464903 18720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:33.466306 18720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:33.466414 18720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:33.466564 18720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:33.466639 18720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:33.501399 18720 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:43757
I20260810 23:32:33.501448 18935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:43757 every 8 connection(s)
I20260810 23:32:33.502920 18720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:32:33.512931 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 18720
I20260810 23:32:33.513149 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:32:33.514020 9411 external_mini_cluster.cc:1428] Setting key 230c2d5c19adcd74a65d2d51d6508d43
Aug 10 23:32:33 dist-test-slave-04bb krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404753, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:33.525730 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:33.505664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:55537 (local address 127.9.48.254:41791)
0810 23:32:33.506139 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:33.506144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:33.507053 (+ 909us) server_negotiation.cc:407] Connection header received
0810 23:32:33.508983 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:33.508989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:33.509100 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:33.509227 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:33.510749 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:33.511555 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:33.513349 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:33.514083 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:33.514237 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:33.518367 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:33.518377 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:33.518385 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:33.518434 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:33.522190 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:33.523021 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:33.523029 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:33.523035 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:33.523197 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:33.523636 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:33.523644 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:33.523649 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:33.524054 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:33.524210 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:33.524946 (+ 736us) server_negotiation.cc:299] Negotiation successful
0810 23:32:33.525232 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":253,"threads_started":1}
I20260810 23:32:33.526403 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:33.506144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:41791 (local address 127.9.48.193:55537)
0810 23:32:33.506889 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:33.506934 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:33.508715 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:33.509475 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:33.509484 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:33.509984 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:33.510552 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:33.510569 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:33.511722 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:33.511727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:33.513135 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:33.513148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:33.514317 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:33.514323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:33.514510 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:33.515329 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:33.515357 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:33.518130 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:33.522398 (+ 4268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:33.522406 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:33.522419 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:33.522881 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:33.523322 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:33.523328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:33.523331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:33.523516 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:33.524180 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:33.524186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:33.524492 (+ 306us) client_negotiation.cc:769] Sending connection context
0810 23:32:33.524814 (+ 322us) client_negotiation.cc:240] Negotiation successful
0810 23:32:33.525208 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":308,"threads_started":1}
I20260810 23:32:33.528162 18936 heartbeater.cc:347] Connected to a master server at 127.9.48.254:41791
I20260810 23:32:33.528486 18936 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:33.529412 18936 heartbeater.cc:511] Master 127.9.48.254:41791 requested a full tablet report, sending...
I20260810 23:32:33.531715 18628 ts_manager.cc:194] Registered new tserver with Master: e9c55c2bf2dd4cd6b54084816fd65e7f (127.9.48.193:43757)
I20260810 23:32:33.533417 18628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:55537
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:32:33.573930 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41791
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:33.914028 18946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:33.914359 18946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:33.914538 18946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:33.921677 18946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:33.921823 18946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:33.922012 18946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:32:33.931592 18946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:33.932891 18946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:33.934227 18946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:33.947675 18956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:33.947834 18953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:33.948460 18959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:33.950256 18946 server_base.cc:1034] running on GCE node
I20260810 23:32:33.950922 18946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:33.951861 18946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:33.953087 18946 hybrid_clock.cc:648] HybridClock initialized: now 1786404753953052 us; error 31 us; skew 500 ppm
I20260810 23:32:33.953449 18946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:33 dist-test-slave-04bb krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404753, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:33.956976 18946 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:32:33.958619 18946 webserver.cc:460] Webserver started at http://127.9.48.194:46247/ using document root <none> and password file <none>
I20260810 23:32:33.959432 18946 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:33.959591 18946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:33.959929 18946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:33.963156 18946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "a6a8b956de1241e79fa4ac5242b4b839"
format_stamp: "Formatted at 2026-08-10 23:32:33 on dist-test-slave-04bb"
server_key: "b37d60851a6a8e06b6da6e633b420572"
server_key_iv: "5ac5224a99ed632435d817b21c7dee62"
server_key_version: "encryptionkey@0"
I20260810 23:32:33.964000 18946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "a6a8b956de1241e79fa4ac5242b4b839"
format_stamp: "Formatted at 2026-08-10 23:32:33 on dist-test-slave-04bb"
server_key: "b37d60851a6a8e06b6da6e633b420572"
server_key_iv: "5ac5224a99ed632435d817b21c7dee62"
server_key_version: "encryptionkey@0"
I20260810 23:32:33.969748 18946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 23:32:33.973376 18969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:33.974776 18946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:32:33.974956 18946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "a6a8b956de1241e79fa4ac5242b4b839"
format_stamp: "Formatted at 2026-08-10 23:32:33 on dist-test-slave-04bb"
server_key: "b37d60851a6a8e06b6da6e633b420572"
server_key_iv: "5ac5224a99ed632435d817b21c7dee62"
server_key_version: "encryptionkey@0"
I20260810 23:32:33.975159 18946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:33.991402 18946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:33.995612 18946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:33.995918 18946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:33.996977 18946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:33.998327 18946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:33.998430 18946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:33.998588 18946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:33.998672 18946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:34.033347 18946 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:32899
I20260810 23:32:34.033386 19142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:32899 every 8 connection(s)
I20260810 23:32:34.034633 18946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:32:34.040777 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 18946
I20260810 23:32:34.040992 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:32:34.041496 9411 external_mini_cluster.cc:1428] Setting key 99574aaf3040a42c9cf0444911682f58
Aug 10 23:32:34 dist-test-slave-04bb krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404753, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:34.057374 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:34.037351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:45411 (local address 127.9.48.254:41791)
0810 23:32:34.037777 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:34.037782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:34.038829 (+ 1047us) server_negotiation.cc:407] Connection header received
0810 23:32:34.040704 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:34.040710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:34.040818 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:34.040960 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:34.042716 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.043753 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.045360 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.046124 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.046272 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:34.050124 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:34.050133 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:34.050141 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:34.050185 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:34.053898 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.054915 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.054919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.054921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.055058 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.055595 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.055599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.055601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.055885 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:34.056000 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:34.056910 (+ 910us) server_negotiation.cc:299] Negotiation successful
0810 23:32:34.057077 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":314,"thread_start_us":204,"threads_started":1}
I20260810 23:32:34.058876 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:34.037915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:41791 (local address 127.9.48.194:45411)
0810 23:32:34.038654 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:34.038696 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:34.040357 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:34.041261 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:34.041269 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:34.041846 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:34.042550 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:34.042570 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.043907 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.043911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:34.045213 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:34.045225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.046408 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.046416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:34.046623 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:34.047523 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:34.047559 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:34.049896 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:34.054061 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:34.054073 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:34.054091 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:34.054759 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:34.055184 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:34.055190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:34.055195 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:34.055469 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:34.056018 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:34.056028 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:34.056479 (+ 451us) client_negotiation.cc:769] Sending connection context
0810 23:32:34.056818 (+ 339us) client_negotiation.cc:240] Negotiation successful
0810 23:32:34.057387 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":309,"threads_started":1}
I20260810 23:32:34.060384 19143 heartbeater.cc:347] Connected to a master server at 127.9.48.254:41791
I20260810 23:32:34.060736 19143 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:34.061471 19143 heartbeater.cc:511] Master 127.9.48.254:41791 requested a full tablet report, sending...
I20260810 23:32:34.063000 18628 ts_manager.cc:194] Registered new tserver with Master: a6a8b956de1241e79fa4ac5242b4b839 (127.9.48.194:32899)
I20260810 23:32:34.063841 18628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:45411
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:32:34.102264 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41791
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:34.440114 19158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:34.440382 19158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:34.440589 19158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:34.448724 19158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:34.448822 19158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:34.448988 19158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:32:34.458673 19158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:41791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:34.459981 19158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:34.461375 19158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:34.475025 19176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:34.475111 19166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:34.475173 19172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:34.475919 19158 server_base.cc:1034] running on GCE node
I20260810 23:32:34.476617 19158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:34.477586 19158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:34.478796 19158 hybrid_clock.cc:648] HybridClock initialized: now 1786404754478748 us; error 36 us; skew 500 ppm
I20260810 23:32:34.479125 19158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:34 dist-test-slave-04bb krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404754, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:34.482595 19158 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:32:34.484136 19158 webserver.cc:460] Webserver started at http://127.9.48.195:39733/ using document root <none> and password file <none>
I20260810 23:32:34.484997 19158 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:34.485175 19158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:34.485491 19158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:34.488399 19158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "7e2473b2a6384850a963a9c2f42dc1f5"
format_stamp: "Formatted at 2026-08-10 23:32:34 on dist-test-slave-04bb"
server_key: "a5d52be1e6e356eafba7ae05a2224952"
server_key_iv: "c25618fa5f097633406955a5fd547a14"
server_key_version: "encryptionkey@0"
I20260810 23:32:34.489250 19158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "7e2473b2a6384850a963a9c2f42dc1f5"
format_stamp: "Formatted at 2026-08-10 23:32:34 on dist-test-slave-04bb"
server_key: "a5d52be1e6e356eafba7ae05a2224952"
server_key_iv: "c25618fa5f097633406955a5fd547a14"
server_key_version: "encryptionkey@0"
I20260810 23:32:34.494431 19158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:32:34.497925 19184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:34.499256 19158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:32:34.499441 19158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "7e2473b2a6384850a963a9c2f42dc1f5"
format_stamp: "Formatted at 2026-08-10 23:32:34 on dist-test-slave-04bb"
server_key: "a5d52be1e6e356eafba7ae05a2224952"
server_key_iv: "c25618fa5f097633406955a5fd547a14"
server_key_version: "encryptionkey@0"
I20260810 23:32:34.499679 19158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:34.516113 19158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:34.520509 19158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:34.520848 19158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:34.521824 19158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:34.523201 19158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:34.523307 19158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:34.523463 19158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:34.523538 19158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:34.537307 18936 heartbeater.cc:503] Master 127.9.48.254:41791 was elected leader, sending a full tablet report...
I20260810 23:32:34.559195 19158 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:33847
I20260810 23:32:34.559280 19346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:33847 every 8 connection(s)
I20260810 23:32:34.560755 19158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:32:34.566924 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 19158
I20260810 23:32:34.567126 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:32:34.567602 9411 external_mini_cluster.cc:1428] Setting key 8fff01cbccc97cc0d18d842f88086378
Aug 10 23:32:34 dist-test-slave-04bb krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404754, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:34.581729 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:34.563259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:38031 (local address 127.9.48.254:41791)
0810 23:32:34.563686 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:34.563690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:34.564831 (+ 1141us) server_negotiation.cc:407] Connection header received
0810 23:32:34.566655 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:34.566661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:34.566792 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:34.566943 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:34.569007 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.570171 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.571993 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.572417 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.572515 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:34.575916 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:34.575920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:34.575924 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:34.575955 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:34.578547 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.579217 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.579221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.579223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.579352 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.579787 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.579790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.579792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.580154 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:34.580274 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:34.581027 (+ 753us) server_negotiation.cc:299] Negotiation successful
0810 23:32:34.581266 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":202,"threads_started":1}
I20260810 23:32:34.582505 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:34.563812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:41791 (local address 127.9.48.195:38031)
0810 23:32:34.564667 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:34.564712 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:34.566369 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:34.567264 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:34.567276 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:34.568044 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:34.568835 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:34.568853 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.570360 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.570366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:34.571856 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:34.571870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.572664 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.572669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:34.572878 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:34.573589 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:34.573612 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:34.575727 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:34.578675 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:34.578681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:34.578692 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:34.579117 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:34.579450 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:34.579454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:34.579456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:34.579674 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:34.580262 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:34.580267 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:34.580640 (+ 373us) client_negotiation.cc:769] Sending connection context
0810 23:32:34.580867 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 23:32:34.581222 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":347,"threads_started":1}
I20260810 23:32:34.583864 19349 heartbeater.cc:347] Connected to a master server at 127.9.48.254:41791
I20260810 23:32:34.584204 19349 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:34.585075 19349 heartbeater.cc:511] Master 127.9.48.254:41791 requested a full tablet report, sending...
I20260810 23:32:34.586625 18626 ts_manager.cc:194] Registered new tserver with Master: 7e2473b2a6384850a963a9c2f42dc1f5 (127.9.48.195:33847)
I20260810 23:32:34.587589 18626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:38031
I20260810 23:32:34.595705 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/test-admin.keytab.
Aug 10 23:32:34 dist-test-slave-04bb krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:34.630205 9411 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 23:32:34.635464 19361 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:34.632993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.9.48.193:43757)
0810 23:32:34.633680 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:34.633688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:34.633716 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:32:34.633780 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:34.633787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:34.634012 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:34.634147 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:34.635002 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42028: BlockingRecv error: recv got EOF from 127.0.0.1:42028 (error 108)
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":365,"threads_started":1}
W20260810 23:32:34.635931 9411 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/krb5cc)
Aug 10 23:32:34 dist-test-slave-04bb krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:34.637828 9411 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:32:34 dist-test-slave-04bb krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:34.639241 9411 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:32:34 dist-test-slave-04bb krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:32:34.654140 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:34.640835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.9.48.193:43757)
0810 23:32:34.641044 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:34.641050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:34.641316 (+ 266us) server_negotiation.cc:407] Connection header received
0810 23:32:34.641451 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:34.641455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:34.641513 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:34.641594 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:34.642494 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.643434 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.644649 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.645191 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.645290 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:34.648966 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:34.648977 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:34.648992 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:34.649022 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:34.651626 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.652221 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.652228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.652230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.652354 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.652765 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.652769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.652771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.653104 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:34.653329 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:34.653599 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 23:32:34.653887 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 23:32:34.660455 19361 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:34.658530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.9.48.193:43757)
0810 23:32:34.658743 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:34.658748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:34.658962 (+ 214us) server_negotiation.cc:407] Connection header received
0810 23:32:34.659171 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:34.659175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:34.659244 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:34.659359 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:34.660162 (+ 803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42042: BlockingRecv error: recv got EOF from 127.0.0.1:42042 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 23:32:34.660785 9411 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/krb5cc' of type FILE
Aug 10 23:32:34 dist-test-slave-04bb krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:34.662238 9411 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:32:34 dist-test-slave-04bb krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:34.663561 9411 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:32:34 dist-test-slave-04bb krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:32:34.678963 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:34.664853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.9.48.193:43757)
0810 23:32:34.665017 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:34.665025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:34.665423 (+ 398us) server_negotiation.cc:407] Connection header received
0810 23:32:34.665596 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:34.665602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:34.665664 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:34.665766 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:34.666715 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.667559 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.668901 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.669341 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.669482 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:34.673010 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:34.673016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:34.673020 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:34.673072 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:34.676009 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.676866 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.676874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.676878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.677032 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.677451 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.677459 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.677467 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.677741 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:34.678050 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:34.678439 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 23:32:34.678655 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 23:32:34.683720 19361 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:34.681958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.9.48.193:43757)
0810 23:32:34.682169 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:34.682178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:34.682467 (+ 289us) server_negotiation.cc:407] Connection header received
0810 23:32:34.682558 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:34.682562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:34.682631 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:34.682716 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:34.683445 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42070: BlockingRecv error: recv got EOF from 127.0.0.1:42070 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 23:32:34.683833 9411 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786404698557218-9411-0/krb5kdc/krb5cc)
Aug 10 23:32:34 dist-test-slave-04bb krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:34.685428 9411 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:32:34 dist-test-slave-04bb krb5kdc[18564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:34.687124 9411 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:32:34 dist-test-slave-04bb krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:32:34.700879 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:34.688501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.9.48.193:43757)
0810 23:32:34.688712 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:34.688717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:34.688869 (+ 152us) server_negotiation.cc:407] Connection header received
0810 23:32:34.688972 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:34.688976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:34.689028 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:34.689105 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:34.690062 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.690768 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.691941 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.692364 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.692459 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:34.696198 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:34.696203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:34.696207 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:34.696235 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:34.698644 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.699222 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.699225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.699228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.699367 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.699680 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.699683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.699685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.699933 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:34.700149 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:34.700349 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 23:32:34.700524 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:32:34.702543 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 18720
I20260810 23:32:34.714874 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:34.703300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.9.48.193:43757)
0810 23:32:34.703567 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:34.703577 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:34.703930 (+ 353us) server_negotiation.cc:407] Connection header received
0810 23:32:34.704089 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:34.704094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:34.704160 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:34.704259 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:34.705368 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.706024 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.707385 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.708034 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.708166 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:34.710131 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:34.710137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:34.710141 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:34.710179 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:34.712363 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.713022 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.713027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.713031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.713165 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.713560 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.713565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.713568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.713814 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:34.714068 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:34.714374 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 23:32:34.714567 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:32:34.715456 18927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:34.801671 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 18720
I20260810 23:32:34.828123 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 18946
Aug 10 23:32:34 dist-test-slave-04bb krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:32:34.843317 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:34.829075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.9.48.194:32899)
0810 23:32:34.829687 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:34.829694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:34.829721 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:32:34.829793 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:34.829800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:34.830009 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:34.830145 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:34.831130 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.832146 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.833457 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.834004 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.834101 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:34.837965 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:34.837977 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:34.837990 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:34.838021 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:34.840790 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.841412 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.841419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.841422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.841547 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.841905 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.841909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.841910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.842238 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:34.842483 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:34.842733 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 23:32:34.842985 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":323,"threads_started":1}
I20260810 23:32:34.844668 19138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:34.928184 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 18946
I20260810 23:32:34.950623 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 19158
Aug 10 23:32:34 dist-test-slave-04bb krb5kdc[18564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:32:34.967669 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:34.951712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49036 (local address 127.9.48.195:33847)
0810 23:32:34.952496 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:34.952504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:34.952608 (+ 104us) server_negotiation.cc:407] Connection header received
0810 23:32:34.952692 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:34.952700 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:34.952944 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:34.953153 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:34.954375 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.955519 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.957102 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:34.957690 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:34.957799 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:34.962091 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:34.962102 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:34.962116 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:34.962149 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:34.965004 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.965699 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.965716 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.965719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.965845 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:34.966248 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:34.966251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:34.966253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:34.966597 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:34.966931 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:34.967159 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 23:32:34.967378 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":396,"threads_started":1}
I20260810 23:32:34.969195 19338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:35.052561 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 19158
I20260810 23:32:35.076680 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 18582
I20260810 23:32:35.078006 18681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:35.182724 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 18582
2026-08-10T23:32:35Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (5123 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:32:35 dist-test-slave-04bb krb5kdc[19390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:32:35 dist-test-slave-04bb krb5kdc[19390](info): set up 2 sockets
Aug 10 23:32:35 dist-test-slave-04bb krb5kdc[19390](info): commencing operation
krb5kdc: starting...
W20260810 23:32:36.530923 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.295s user 0.004s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:32:36 dist-test-slave-04bb krb5kdc[19390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:32:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.254.keytab.
Entry for principal oryx/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.254.keytab.
I20260810 23:32:36.772830 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:40553
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.254.keytab
--principal=oryx/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:40553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:37.112854 19405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:37.113204 19405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:37.113358 19405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:37.120435 19405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:37.120607 19405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:37.120718 19405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:37.120822 19405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:37.120899 19405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:37.130852 19405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:40553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.254.keytab
--principal=oryx/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:40553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:37.132215 19405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:37.133667 19405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:37.145705 19417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:37.145844 19419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:37.145838 19416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:37.146822 19405 server_base.cc:1034] running on GCE node
I20260810 23:32:37.147632 19405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:37.149166 19405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:37.150389 19405 hybrid_clock.cc:648] HybridClock initialized: now 1786404757150337 us; error 51 us; skew 500 ppm
I20260810 23:32:37.150725 19405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:37 dist-test-slave-04bb krb5kdc[19390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404757, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:37.153970 19405 init.cc:377] Logged in from keytab as oryx/127.9.48.254@KRBTEST.COM (short username oryx)
I20260810 23:32:37.155498 19405 webserver.cc:460] Webserver started at http://127.9.48.254:44597/ using document root <none> and password file <none>
I20260810 23:32:37.156229 19405 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:37.156379 19405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:37.156766 19405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:37.159793 19405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "2724aeacf453402d98aecdbb20891190"
format_stamp: "Formatted at 2026-08-10 23:32:37 on dist-test-slave-04bb"
server_key: "371c0e5f90dd6c75e49f8f7194af090d"
server_key_iv: "ff0f6567e86edeec6d4ae4c5da610a55"
server_key_version: "encryptionkey@0"
I20260810 23:32:37.160692 19405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "2724aeacf453402d98aecdbb20891190"
format_stamp: "Formatted at 2026-08-10 23:32:37 on dist-test-slave-04bb"
server_key: "371c0e5f90dd6c75e49f8f7194af090d"
server_key_iv: "ff0f6567e86edeec6d4ae4c5da610a55"
server_key_version: "encryptionkey@0"
I20260810 23:32:37.165964 19405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:32:37.169436 19432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:37.170823 19405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:32:37.171010 19405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "2724aeacf453402d98aecdbb20891190"
format_stamp: "Formatted at 2026-08-10 23:32:37 on dist-test-slave-04bb"
server_key: "371c0e5f90dd6c75e49f8f7194af090d"
server_key_iv: "ff0f6567e86edeec6d4ae4c5da610a55"
server_key_version: "encryptionkey@0"
I20260810 23:32:37.171216 19405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:37.209592 19405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:37.214358 19405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:37.214689 19405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:37.234104 19405 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:40553
I20260810 23:32:37.234141 19518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:40553 every 8 connection(s)
I20260810 23:32:37.235525 19405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:37.236802 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 19405
I20260810 23:32:37.237035 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:32:37.237890 9411 external_mini_cluster.cc:1428] Setting key 1d362475baf7465fceb5a55bbe852327
I20260810 23:32:37.240703 19519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:32:37 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
I20260810 23:32:37.254086 19519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190: Bootstrap starting.
I20260810 23:32:37.257301 19519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:37.258342 19519 log.cc:824] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:37.261494 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:37.240068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.9.48.254:40553)
0810 23:32:37.240961 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:37.240977 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:37.241026 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:32:37.242615 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:37.242636 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:37.243010 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:37.243347 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:37.245128 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:37.246467 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:37.248280 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:37.248999 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:37.249123 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:37.254007 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:37.254015 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:37.254029 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:37.254064 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:37.257168 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:37.257929 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:37.257936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:37.257942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:37.258116 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:37.258506 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:37.258513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:37.258516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:37.259381 (+ 865us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:37.259728 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:37.260151 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 23:32:37.260449 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":369,"threads_started":1}
I20260810 23:32:37.261886 19519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190: No bootstrap required, opened a new log
I20260810 23:32:37.270000 19519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } }
I20260810 23:32:37.270385 19519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:37.270521 19519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2724aeacf453402d98aecdbb20891190, State: Initialized, Role: FOLLOWER
I20260810 23:32:37.271176 19519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } }
I20260810 23:32:37.271459 19519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:32:37.271620 19519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:32:37.271826 19519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:37.274175 19519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } }
I20260810 23:32:37.274726 19519 leader_election.cc:302] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2724aeacf453402d98aecdbb20891190; no voters:
I20260810 23:32:37.275496 19519 leader_election.cc:288] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:32:37.275687 19525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:37.276580 19525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [term 1 LEADER]: Becoming Leader. State: Replica: 2724aeacf453402d98aecdbb20891190, State: Running, Role: LEADER
I20260810 23:32:37.277190 19525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } }
I20260810 23:32:37.277832 19519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:37.280238 19528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2724aeacf453402d98aecdbb20891190. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } }
I20260810 23:32:37.280762 19528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:37.280582 19527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } }
I20260810 23:32:37.280905 19527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:37.282496 19542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:37.287093 19542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:37.295442 19542 catalog_manager.cc:1347] Generated new cluster ID: e341c7ce902047f2843c6a47b0eb866a
I20260810 23:32:37.295550 19542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:37.313174 19542 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:32:37.315179 19542 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:37.328505 19542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190: Generated new TSK 0
I20260810 23:32:37.329581 19542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.193.keytab.
Entry for principal oryx/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.193.keytab.
I20260810 23:32:37.397385 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40553
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.193.keytab
--principal=oryx/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:37.755895 19557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:37.756145 19557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:37.756313 19557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:37.763691 19557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:37.763792 19557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:37.763979 19557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:32:37.774544 19557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.193.keytab
--principal=oryx/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:37.775966 19557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:37.777357 19557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:37.790874 19564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:37.791090 19565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:37.791155 19568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:37.792109 19557 server_base.cc:1034] running on GCE node
I20260810 23:32:37.792785 19557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:37.793727 19557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:37.794927 19557 hybrid_clock.cc:648] HybridClock initialized: now 1786404757794874 us; error 56 us; skew 500 ppm
I20260810 23:32:37.795251 19557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:37 dist-test-slave-04bb krb5kdc[19390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404757, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:37.802548 19557 init.cc:377] Logged in from keytab as oryx/127.9.48.193@KRBTEST.COM (short username oryx)
I20260810 23:32:37.804716 19557 webserver.cc:460] Webserver started at http://127.9.48.193:35211/ using document root <none> and password file <none>
I20260810 23:32:37.805688 19557 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:37.805836 19557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:37.806241 19557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:37.809839 19557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "c94048588350439e91217f75d0058224"
format_stamp: "Formatted at 2026-08-10 23:32:37 on dist-test-slave-04bb"
server_key: "96c7abf6a75771e9fc418a4d67c53070"
server_key_iv: "cf9153dbcae858fb1080cc15600e556b"
server_key_version: "encryptionkey@0"
I20260810 23:32:37.810752 19557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "c94048588350439e91217f75d0058224"
format_stamp: "Formatted at 2026-08-10 23:32:37 on dist-test-slave-04bb"
server_key: "96c7abf6a75771e9fc418a4d67c53070"
server_key_iv: "cf9153dbcae858fb1080cc15600e556b"
server_key_version: "encryptionkey@0"
I20260810 23:32:37.816407 19557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 23:32:37.819816 19577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:37.821252 19557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:37.821439 19557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "c94048588350439e91217f75d0058224"
format_stamp: "Formatted at 2026-08-10 23:32:37 on dist-test-slave-04bb"
server_key: "96c7abf6a75771e9fc418a4d67c53070"
server_key_iv: "cf9153dbcae858fb1080cc15600e556b"
server_key_version: "encryptionkey@0"
I20260810 23:32:37.821643 19557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:37.844918 19557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:37.849406 19557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:37.849717 19557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:37.850718 19557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:37.852171 19557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:37.852290 19557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:37.852450 19557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:37.852527 19557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:37.887362 19557 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:46437
I20260810 23:32:37.887456 19760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:46437 every 8 connection(s)
I20260810 23:32:37.888830 19557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:32:37.895342 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 19557
I20260810 23:32:37.895586 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:32:37.896268 9411 external_mini_cluster.cc:1428] Setting key bced81dc8d7d5bc3d66ba0674def1a5a
Aug 10 23:32:37 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404757, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
I20260810 23:32:37.909833 19763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:37.891534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:49015 (local address 127.9.48.254:40553)
0810 23:32:37.892005 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:37.892012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:37.893096 (+ 1084us) server_negotiation.cc:407] Connection header received
0810 23:32:37.895081 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:37.895095 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:37.895205 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:37.895369 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:37.897156 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:37.898053 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:37.899729 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:37.900143 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:37.900299 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:37.904010 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:37.904015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:37.904018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:37.904052 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:37.906634 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:37.907389 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:37.907401 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:37.907403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:37.907532 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:37.908008 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:37.908012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:37.908014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:37.908284 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:37.908448 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:37.909221 (+ 773us) server_negotiation.cc:299] Negotiation successful
0810 23:32:37.909451 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":210,"threads_started":1}
I20260810 23:32:37.910817 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:37.892174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.193:49015)
0810 23:32:37.892937 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:37.892976 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:37.894728 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:37.895662 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:37.895674 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:37.896206 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:37.896952 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:37.896975 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:37.898215 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:37.898225 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:37.899564 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:37.899577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:37.900418 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:37.900427 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:37.900734 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:37.901543 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:37.901567 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:37.903783 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:37.906779 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:37.906788 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:37.906805 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:37.907246 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:37.907648 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:37.907653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:37.907657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:37.907874 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:37.908428 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:37.908436 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:37.908832 (+ 396us) client_negotiation.cc:769] Sending connection context
0810 23:32:37.909101 (+ 269us) client_negotiation.cc:240] Negotiation successful
0810 23:32:37.909643 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":331,"threads_started":1}
I20260810 23:32:37.912604 19761 heartbeater.cc:347] Connected to a master server at 127.9.48.254:40553
I20260810 23:32:37.912930 19761 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:37.913748 19761 heartbeater.cc:511] Master 127.9.48.254:40553 requested a full tablet report, sending...
I20260810 23:32:37.915959 19466 ts_manager.cc:194] Registered new tserver with Master: c94048588350439e91217f75d0058224 (127.9.48.193:46437)
I20260810 23:32:37.917484 19466 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:49015
WARNING: no policy specified for oryx/127.9.48.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.194.keytab.
Entry for principal oryx/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.194.keytab.
I20260810 23:32:37.957788 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40553
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.194.keytab
--principal=oryx/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:38.299729 19768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:38.300082 19768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:38.300246 19768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:38.307566 19768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:38.307719 19768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:38.307901 19768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:32:38.317698 19768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.194.keytab
--principal=oryx/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:38.318994 19768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:38.320389 19768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:38.333753 19774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:38.333729 19773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:38.334318 19780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:38.334658 19768 server_base.cc:1034] running on GCE node
I20260810 23:32:38.335322 19768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:38.336266 19768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:38.337467 19768 hybrid_clock.cc:648] HybridClock initialized: now 1786404758337424 us; error 59 us; skew 500 ppm
I20260810 23:32:38.337781 19768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:38 dist-test-slave-04bb krb5kdc[19390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404758, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:38.340914 19768 init.cc:377] Logged in from keytab as oryx/127.9.48.194@KRBTEST.COM (short username oryx)
I20260810 23:32:38.342422 19768 webserver.cc:460] Webserver started at http://127.9.48.194:40773/ using document root <none> and password file <none>
I20260810 23:32:38.343123 19768 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:38.343271 19768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:38.343603 19768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:38.346560 19768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "2d7651a6bfd3488d88b587ed9db9ee90"
format_stamp: "Formatted at 2026-08-10 23:32:38 on dist-test-slave-04bb"
server_key: "8194dbf54e1b18e260739d6f27a02241"
server_key_iv: "bfd4fd3c924475c8d80d2617126a975f"
server_key_version: "encryptionkey@0"
I20260810 23:32:38.347347 19768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "2d7651a6bfd3488d88b587ed9db9ee90"
format_stamp: "Formatted at 2026-08-10 23:32:38 on dist-test-slave-04bb"
server_key: "8194dbf54e1b18e260739d6f27a02241"
server_key_iv: "bfd4fd3c924475c8d80d2617126a975f"
server_key_version: "encryptionkey@0"
I20260810 23:32:38.352625 19768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:32:38.356026 19792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:38.357329 19768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:38.357496 19768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "2d7651a6bfd3488d88b587ed9db9ee90"
format_stamp: "Formatted at 2026-08-10 23:32:38 on dist-test-slave-04bb"
server_key: "8194dbf54e1b18e260739d6f27a02241"
server_key_iv: "bfd4fd3c924475c8d80d2617126a975f"
server_key_version: "encryptionkey@0"
I20260810 23:32:38.357712 19768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:38.371657 19768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:38.375654 19768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:38.375947 19768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:38.376933 19768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:38.378271 19768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:38.378367 19768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:38.378499 19768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:38.378561 19768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:38.415670 19768 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:43199
I20260810 23:32:38.415722 19991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:43199 every 8 connection(s)
I20260810 23:32:38.417150 19768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:32:38.424979 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 19768
I20260810 23:32:38.425180 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:32:38.425697 9411 external_mini_cluster.cc:1428] Setting key abbef1df643132c84a59b7450d8a086b
Aug 10 23:32:38 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404758, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
I20260810 23:32:38.437279 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:38.419923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:36397 (local address 127.9.48.254:40553)
0810 23:32:38.420393 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:38.420398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:38.421409 (+ 1011us) server_negotiation.cc:407] Connection header received
0810 23:32:38.423216 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:38.423221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:38.423309 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:38.423427 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:38.424986 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:38.425908 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:38.427556 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:38.428024 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:38.428173 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:38.431547 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:38.431556 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:38.431565 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:38.431595 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:38.434289 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:38.434999 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:38.435005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:38.435010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:38.435146 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:38.435589 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:38.435594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:38.435596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:38.435840 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:38.435993 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:38.436661 (+ 668us) server_negotiation.cc:299] Negotiation successful
0810 23:32:38.436873 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":247,"threads_started":1}
I20260810 23:32:38.438181 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:38.420477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.194:36397)
0810 23:32:38.421250 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:38.421294 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:38.422943 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:38.423674 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:38.423682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:38.424175 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:38.424795 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:38.424833 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:38.426068 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:38.426073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:38.427409 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:38.427423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:38.428249 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:38.428257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:38.428482 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:38.429226 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:38.429247 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:38.431337 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:38.434429 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:38.434438 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:38.434452 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:38.434870 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:38.435267 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:38.435273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:38.435277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:38.435460 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:38.435969 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:38.435977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:38.436288 (+ 311us) client_negotiation.cc:769] Sending connection context
0810 23:32:38.436566 (+ 278us) client_negotiation.cc:240] Negotiation successful
0810 23:32:38.437061 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":274,"threads_started":1}
I20260810 23:32:38.439685 19992 heartbeater.cc:347] Connected to a master server at 127.9.48.254:40553
I20260810 23:32:38.439970 19992 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:38.440727 19992 heartbeater.cc:511] Master 127.9.48.254:40553 requested a full tablet report, sending...
I20260810 23:32:38.442189 19466 ts_manager.cc:194] Registered new tserver with Master: 2d7651a6bfd3488d88b587ed9db9ee90 (127.9.48.194:43199)
I20260810 23:32:38.443490 19466 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:36397
WARNING: no policy specified for oryx/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.195.keytab.
Entry for principal oryx/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.195.keytab.
I20260810 23:32:38.484028 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40553
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.195.keytab
--principal=oryx/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:38.822594 20004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:38.822891 20004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:38.822992 20004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:38.830297 20004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:38.830400 20004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:38.830509 20004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:32:38.844935 20004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.195.keytab
--principal=oryx/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:38.846449 20004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:38.848405 20004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:38.863248 20012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:38.863317 20015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:38.863349 20011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:38.864241 20004 server_base.cc:1034] running on GCE node
I20260810 23:32:38.865010 20004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:38.865978 20004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:38.867230 20004 hybrid_clock.cc:648] HybridClock initialized: now 1786404758867200 us; error 77 us; skew 500 ppm
I20260810 23:32:38.867555 20004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:38 dist-test-slave-04bb krb5kdc[19390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404758, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:38.870784 20004 init.cc:377] Logged in from keytab as oryx/127.9.48.195@KRBTEST.COM (short username oryx)
I20260810 23:32:38.872169 20004 webserver.cc:460] Webserver started at http://127.9.48.195:45827/ using document root <none> and password file <none>
I20260810 23:32:38.872939 20004 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:38.873085 20004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:38.873399 20004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:38.876916 20004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "47c185c80b264d0593a81622c4ef6847"
format_stamp: "Formatted at 2026-08-10 23:32:38 on dist-test-slave-04bb"
server_key: "dcd5966dfe19b81c32e3939166e0906b"
server_key_iv: "ee2be83e649b6d3256f1d82fa66e7c2f"
server_key_version: "encryptionkey@0"
I20260810 23:32:38.877691 20004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "47c185c80b264d0593a81622c4ef6847"
format_stamp: "Formatted at 2026-08-10 23:32:38 on dist-test-slave-04bb"
server_key: "dcd5966dfe19b81c32e3939166e0906b"
server_key_iv: "ee2be83e649b6d3256f1d82fa66e7c2f"
server_key_version: "encryptionkey@0"
I20260810 23:32:38.883195 20004 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:32:38.886667 20029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:38.888383 20004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:32:38.888577 20004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "47c185c80b264d0593a81622c4ef6847"
format_stamp: "Formatted at 2026-08-10 23:32:38 on dist-test-slave-04bb"
server_key: "dcd5966dfe19b81c32e3939166e0906b"
server_key_iv: "ee2be83e649b6d3256f1d82fa66e7c2f"
server_key_version: "encryptionkey@0"
I20260810 23:32:38.888796 20004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:38.910789 20004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:38.915059 20004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:38.915310 20004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:38.916204 20004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:38.917631 20004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:38.917711 20004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:38.917795 20004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:38.917840 20004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:38.920419 19761 heartbeater.cc:503] Master 127.9.48.254:40553 was elected leader, sending a full tablet report...
I20260810 23:32:38.952968 20004 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:41079
I20260810 23:32:38.953027 20211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:41079 every 8 connection(s)
I20260810 23:32:38.954278 20004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:32:38.958979 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 20004
I20260810 23:32:38.959169 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:32:38.959801 9411 external_mini_cluster.cc:1428] Setting key f6ffbc47d433923618c9b9bb4ccaba41
Aug 10 23:32:38 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404758, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
I20260810 23:32:38.975692 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:38.956951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:55913 (local address 127.9.48.254:40553)
0810 23:32:38.957474 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:38.957481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:38.958532 (+ 1051us) server_negotiation.cc:407] Connection header received
0810 23:32:38.961169 (+ 2637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:38.961175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:38.961287 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:38.961433 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:38.963530 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:38.964308 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:38.965862 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:38.966254 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:38.966370 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:38.969773 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:38.969778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:38.969781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:38.969811 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:38.972265 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:38.973077 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:38.973081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:38.973084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:38.973205 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:38.973656 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:38.973659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:38.973662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:38.973929 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:38.974123 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:38.974987 (+ 864us) server_negotiation.cc:299] Negotiation successful
0810 23:32:38.975234 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":251,"threads_started":1}
I20260810 23:32:38.976651 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:38.957623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.195:55913)
0810 23:32:38.958355 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:38.958405 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:38.960870 (+ 2465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:38.961855 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:38.961868 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:38.962568 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:38.963344 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:38.963370 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:38.964441 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:38.964444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:38.965717 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:38.965729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:38.966443 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:38.966448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:38.966633 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:38.967267 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:38.967290 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:38.969554 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:38.972415 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:38.972424 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:38.972440 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:38.972936 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:38.973319 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:38.973324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:38.973328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:38.973519 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:38.974078 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:38.974089 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:38.974549 (+ 460us) client_negotiation.cc:769] Sending connection context
0810 23:32:38.974832 (+ 283us) client_negotiation.cc:240] Negotiation successful
0810 23:32:38.975419 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":281,"threads_started":1}
I20260810 23:32:38.978176 20213 heartbeater.cc:347] Connected to a master server at 127.9.48.254:40553
I20260810 23:32:38.978448 20213 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:38.979275 20213 heartbeater.cc:511] Master 127.9.48.254:40553 requested a full tablet report, sending...
I20260810 23:32:38.980721 19466 ts_manager.cc:194] Registered new tserver with Master: 47c185c80b264d0593a81622c4ef6847 (127.9.48.195:41079)
I20260810 23:32:38.981612 19466 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:55913
I20260810 23:32:38.987694 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.48.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.48.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.48.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.253.keytab.
Entry for principal oryx/127.9.48.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.253.keytab.
I20260810 23:32:39.046070 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.253:35727
--webserver_interface=127.9.48.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.253.keytab
--principal=oryx/127.9.48.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:40553,127.9.48.253:35727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:39.393519 20222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:39.393852 20222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:39.393960 20222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:39.401022 20222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:39.401126 20222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:39.401173 20222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:39.401232 20222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:39.401269 20222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:39.410751 20222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:40553,127.9.48.253:35727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.253.keytab
--principal=oryx/127.9.48.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.253:35727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.48.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
W20260810 23:32:39.411059 20222 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.48.254:40553,127.9.48.253:35727'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 23:32:39.412051 20222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:39.413326 20222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:39.425483 20231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:39.425483 20228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:39.425678 20233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:39.426344 20222 server_base.cc:1034] running on GCE node
I20260810 23:32:39.427129 20222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:39.428517 20222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:39.429834 20222 hybrid_clock.cc:648] HybridClock initialized: now 1786404759429768 us; error 34 us; skew 500 ppm
I20260810 23:32:39.430189 20222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:39 dist-test-slave-04bb krb5kdc[19390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404759, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:39.433471 20222 init.cc:377] Logged in from keytab as oryx/127.9.48.253@KRBTEST.COM (short username oryx)
I20260810 23:32:39.434984 20222 webserver.cc:460] Webserver started at http://127.9.48.253:34917/ using document root <none> and password file <none>
I20260810 23:32:39.435714 20222 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:39.435847 20222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:39.436182 20222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:39.439158 20222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/data/instance:
uuid: "ed7ac368f3fe4686b6349f2599e4d93f"
format_stamp: "Formatted at 2026-08-10 23:32:39 on dist-test-slave-04bb"
server_key: "78594f70cb971cb8ef0c11c4f975a0ac"
server_key_iv: "c3a971a99e9c437bd4f022b979227b50"
server_key_version: "encryptionkey@0"
I20260810 23:32:39.439949 20222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/wal/instance:
uuid: "ed7ac368f3fe4686b6349f2599e4d93f"
format_stamp: "Formatted at 2026-08-10 23:32:39 on dist-test-slave-04bb"
server_key: "78594f70cb971cb8ef0c11c4f975a0ac"
server_key_iv: "c3a971a99e9c437bd4f022b979227b50"
server_key_version: "encryptionkey@0"
I20260810 23:32:39.445519 20222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260810 23:32:39.445974 19992 heartbeater.cc:503] Master 127.9.48.254:40553 was elected leader, sending a full tablet report...
I20260810 23:32:39.449167 20246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:39.450510 20222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:32:39.450709 20222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/wal
uuid: "ed7ac368f3fe4686b6349f2599e4d93f"
format_stamp: "Formatted at 2026-08-10 23:32:39 on dist-test-slave-04bb"
server_key: "78594f70cb971cb8ef0c11c4f975a0ac"
server_key_iv: "c3a971a99e9c437bd4f022b979227b50"
server_key_version: "encryptionkey@0"
I20260810 23:32:39.450932 20222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:39.483366 20222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:39.487911 20222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:39.488204 20222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:39.507716 20222 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.253:35727
I20260810 23:32:39.507751 20338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.253:35727 every 8 connection(s)
I20260810 23:32:39.509140 20222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/data/info.pb
I20260810 23:32:39.510298 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 20222
I20260810 23:32:39.513880 20340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:39.520911 20340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:32:39 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.48.253@KRBTEST.COM
Aug 10 23:32:39 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404759, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.253@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
I20260810 23:32:39.534655 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:39.512734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.9.48.253:35727)
0810 23:32:39.513509 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:39.513520 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:39.513562 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:32:39.515220 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:39.515242 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:39.515601 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:39.516001 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:39.517631 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.519100 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.520326 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.521109 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.521333 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:39.526700 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:39.526712 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:39.526725 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:39.526771 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:39.530265 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:39.531127 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:39.531138 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:39.531148 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:39.531419 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:39.531878 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:39.531881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:39.531883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:39.532381 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:39.532723 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:39.533149 (+ 426us) server_negotiation.cc:299] Negotiation successful
0810 23:32:39.533536 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"spinlock_wait_cycles":13184,"thread_start_us":340,"threads_started":1}
I20260810 23:32:39.540174 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:39.523606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.9.48.254:40553)
0810 23:32:39.524090 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:39.524095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:39.524797 (+ 702us) server_negotiation.cc:407] Connection header received
0810 23:32:39.525005 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:39.525009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:39.525111 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:39.525213 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:39.526775 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.527923 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.529605 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.530257 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.530406 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:39.534198 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:39.534203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:39.534207 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:39.534243 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:39.537230 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:39.538078 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:39.538083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:39.538085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:39.538208 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:39.538606 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:39.538610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:39.538612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:39.538928 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:39.539090 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:39.539550 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 23:32:39.539777 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":252,"threads_started":1}
I20260810 23:32:39.540210 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:39.523671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:32930)
0810 23:32:39.524602 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:39.524671 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:39.524858 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:39.525326 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:39.525334 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:39.526211 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:39.526592 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:39.526615 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.528100 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.528104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:39.529450 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:39.529463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.530473 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.530480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:39.530695 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:39.531373 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:39.531395 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:39.534000 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:39.537364 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:39.537377 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:39.537384 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:39.537940 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:39.538326 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:39.538331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:39.538335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:39.538465 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:39.539066 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:39.539073 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:39.539244 (+ 171us) client_negotiation.cc:769] Sending connection context
0810 23:32:39.539443 (+ 199us) client_negotiation.cc:240] Negotiation successful
0810 23:32:39.539768 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":563,"threads_started":1}
I20260810 23:32:39.542026 20340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:32:39 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404759, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.253@KRBTEST.COM for oryx/127.9.48.253@KRBTEST.COM
I20260810 23:32:39.553675 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:39.542870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:35727 (local address 127.0.0.1:37332)
0810 23:32:39.543021 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:39.543036 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:39.543121 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:39.543381 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:39.543384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:39.543686 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:39.543817 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:39.543824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.544816 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.544820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:39.545680 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:39.545690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.546290 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.546294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:39.546414 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:39.546885 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:39.546900 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:39.548854 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:39.551650 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:39.551662 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:39.551664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:39.552023 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:39.552359 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:39.552362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:39.552364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:39.552493 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:39.552976 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:39.552980 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:39.553092 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 23:32:39.553213 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 23:32:39.553407 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 23:32:39.553862 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:39.542810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.9.48.253:35727)
0810 23:32:39.542987 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:39.542993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:39.543101 (+ 108us) server_negotiation.cc:407] Connection header received
0810 23:32:39.543210 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:39.543213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:39.543265 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:39.543346 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:39.543932 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.544688 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.545801 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.546167 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.546255 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:39.549015 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:39.549020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:39.549023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:39.549050 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:39.551520 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:39.552128 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:39.552132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:39.552134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:39.552256 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:39.552631 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:39.552634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:39.552636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:39.552875 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:39.552985 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:39.553298 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 23:32:39.553528 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":55}
I20260810 23:32:39.558956 20340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: Bootstrap starting.
I20260810 23:32:39.562301 20340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:39.563328 20340 log.cc:824] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:39.565905 20340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: No bootstrap required, opened a new log
I20260810 23:32:39.572790 20340 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } }
I20260810 23:32:39.573061 20340 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:39.573184 20340 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7ac368f3fe4686b6349f2599e4d93f, State: Initialized, Role: FOLLOWER
I20260810 23:32:39.573787 20340 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } }
I20260810 23:32:39.574625 20352 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } } }
I20260810 23:32:39.575001 20352 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:39.575693 20340 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:32:39.584913 20367 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:32:39.585050 20367 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:32:39.588735 20222 master_runner.cc:428] Detected that this master ed7ac368f3fe4686b6349f2599e4d93f is joining an existing cluster
I20260810 23:32:39.588886 20222 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.48.253:35727
I20260810 23:32:39.612123 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:39.592738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.9.48.254:40553)
0810 23:32:39.592952 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:39.592960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:39.593640 (+ 680us) server_negotiation.cc:407] Connection header received
0810 23:32:39.593919 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:39.593927 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:39.594034 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:39.594135 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:39.596151 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.597641 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.600348 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.601053 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.601168 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:39.603847 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:39.603855 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:39.603859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:39.603897 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:39.607530 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:39.608915 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:39.608921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:39.608925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:39.609081 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:39.609546 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:39.609551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:39.609554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:39.609930 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:39.610060 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:39.611146 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0810 23:32:39.611388 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:32:39.613363 20384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:39.592742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:32932)
0810 23:32:39.593541 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:39.593564 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:39.593745 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:39.594490 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:39.594498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:39.595381 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:39.595927 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:39.595944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.597889 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.597897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:39.600139 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:39.600160 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.601452 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.601458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:39.601664 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:39.602434 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:39.602461 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:39.603629 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:39.607731 (+ 4102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:39.607736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:39.607741 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:39.608738 (+ 997us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:39.609238 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:39.609243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:39.609248 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:39.609411 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:39.610171 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:39.610178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:39.610727 (+ 549us) client_negotiation.cc:769] Sending connection context
0810 23:32:39.612416 (+ 1689us) client_negotiation.cc:240] Negotiation successful
0810 23:32:39.612833 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":424,"threads_started":1}
I20260810 23:32:39.616644 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:39.593721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37342 (local address 127.9.48.253:35727)
0810 23:32:39.593881 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:39.593886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:39.600932 (+ 7046us) server_negotiation.cc:407] Connection header received
0810 23:32:39.601322 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:39.601330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:39.601398 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:39.601468 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:39.603041 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.604101 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.606204 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.606919 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.607020 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:39.609578 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:39.609584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:39.609590 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:39.609626 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:39.613414 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:39.614401 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:39.614406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:39.614409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:39.614562 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:39.615008 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:39.615012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:39.615015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:39.615402 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:39.615557 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:39.616024 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 23:32:39.616225 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:32:39.617762 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:39.594150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:35727 (local address 127.0.0.1:37342)
0810 23:32:39.600838 (+ 6688us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:39.600858 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:39.601178 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:39.601647 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:39.601651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:39.602509 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:39.602880 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:39.602899 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.604289 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.604294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:39.606053 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:39.606069 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.607335 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.607344 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:39.607524 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:39.608206 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:39.608232 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:39.609411 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:39.613752 (+ 4341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:39.613759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:39.613765 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:39.614275 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:39.614698 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:39.614703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:39.614707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:39.614888 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:39.615635 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:39.615639 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:39.615760 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 23:32:39.616882 (+ 1122us) client_negotiation.cc:240] Negotiation successful
0810 23:32:39.617214 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6559,"thread_start_us":6777,"threads_started":1}
WARNING: no policy specified for oryx/127.9.48.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.48.252@KRBTEST.COM" created.
Aug 10 23:32:39 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404757, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.254@KRBTEST.COM for oryx/127.9.48.253@KRBTEST.COM
I20260810 23:32:39.640722 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:39.620842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.9.48.253:35727)
0810 23:32:39.621034 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:39.621043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:39.624809 (+ 3766us) server_negotiation.cc:407] Connection header received
0810 23:32:39.625197 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:39.625205 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:39.625370 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:39.625480 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:39.627048 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.628213 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.630164 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.630966 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.631121 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:39.635000 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:39.635006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:39.635009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:39.635038 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:39.637866 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:39.638616 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:39.638622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:39.638627 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:39.638766 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:39.639175 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:39.639180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:39.639181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:39.639455 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:39.639620 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:39.640090 (+ 470us) server_negotiation.cc:299] Negotiation successful
0810 23:32:39.640322 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:32:39.640904 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:39.620513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:35727 (local address 127.0.0.1:37346)
0810 23:32:39.624666 (+ 4153us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:39.624712 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:39.624963 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:39.625554 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:39.625565 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:39.626570 (+ 1005us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:39.626862 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:39.626876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.628377 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.628383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:39.629978 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:39.629997 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.631149 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.631156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:39.631389 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:39.632111 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:39.632137 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:39.634810 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:39.638062 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:39.638075 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:39.638083 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:39.638485 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:39.638888 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:39.638894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:39.638898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:39.639047 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:39.639607 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:39.639616 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:39.639791 (+ 175us) client_negotiation.cc:769] Sending connection context
0810 23:32:39.639984 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 23:32:39.640313 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3831,"thread_start_us":3667,"threads_started":1}
I20260810 23:32:39.643122 19466 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.48.253:35727: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f"
member_type: NON_VOTER
last_known_addr {
host: "127.9.48.253"
port: 35727
}
attrs {
promote: true
}
}
dest_uuid: "2724aeacf453402d98aecdbb20891190"
cas_config_opid_index: -1
I20260810 23:32:39.644376 19466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } }
I20260810 23:32:39.653420 20392 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ed7ac368f3fe4686b6349f2599e4d93f (127.9.48.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } } }
I20260810 23:32:39.654459 20283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:39.657804 20283 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 FOLLOWER]: Refusing update from remote peer 2724aeacf453402d98aecdbb20891190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 23:32:39.658856 20395 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:39.659981 20392 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.48.253:35727
I20260810 23:32:39.661221 20222 master.cc:537] Master@127.9.48.253:35727 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.48.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.252.keytab.
Entry for principal oryx/127.9.48.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.252.keytab.
W20260810 23:32:39.664803 19437 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260810 23:32:39.665066 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.252:34131
--webserver_interface=127.9.48.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.252.keytab
--principal=oryx/127.9.48.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:40553,127.9.48.253:35727,127.9.48.252:34131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:39.678409 19437 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 -> Peer ed7ac368f3fe4686b6349f2599e4d93f (127.9.48.253:35727): Couldn't send request to peer ed7ac368f3fe4686b6349f2599e4d93f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:39.681557 20396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } } }
I20260810 23:32:39.682047 20396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:39.684090 20222 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:39.684604 20222 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:39.684829 20222 tablet_replica.cc:330] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: stopping tablet replica
I20260810 23:32:39.692162 20222 master.cc:552] Master@127.9.48.253:35727 shutdown complete.
I20260810 23:32:39.692296 20222 master_runner.cc:305] Clearing existing system tablet
I20260810 23:32:39.692878 20222 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:39.692983 20222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:39.701562 20222 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.000s sys 0.008s
I20260810 23:32:39.703540 20406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:39.703941 20222 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:39.704375 20222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:32:39.704542 20222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/wal
uuid: "ed7ac368f3fe4686b6349f2599e4d93f"
format_stamp: "Formatted at 2026-08-10 23:32:39 on dist-test-slave-04bb"
server_key: "78594f70cb971cb8ef0c11c4f975a0ac"
server_key_iv: "c3a971a99e9c437bd4f022b979227b50"
server_key_version: "encryptionkey@0"
I20260810 23:32:39.710140 20222 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:39.717238 20222 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 23:32:39.717459 20222 log.cc:1196] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 23:32:39.717947 20222 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: Deleting consensus metadata
I20260810 23:32:39.718246 20222 master_runner.cc:315] Copying system tablet from 127.9.48.254:40553
I20260810 23:32:39.721218 20222 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: tablet copy: Beginning tablet copy session from remote peer at address 127.9.48.254:40553
I20260810 23:32:39.738011 20417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:39.722327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:32942)
0810 23:32:39.722885 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:39.722905 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:39.723121 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:39.723739 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:39.723744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:39.724258 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:39.724593 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:39.724607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.726117 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.726123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:39.727683 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:39.727700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.728708 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.728714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:39.728898 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:39.729651 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:39.729674 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:39.730761 (+ 1087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:39.734897 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:39.734902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:39.734906 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:39.735457 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:39.735917 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:39.735925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:39.735930 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:39.736103 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:39.736834 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:39.736840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:39.737042 (+ 202us) client_negotiation.cc:769] Sending connection context
0810 23:32:39.737240 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 23:32:39.737559 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":267,"threads_started":1}
I20260810 23:32:39.738499 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:39.722342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.9.48.254:40553)
0810 23:32:39.722523 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:39.722531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:39.722997 (+ 466us) server_negotiation.cc:407] Connection header received
0810 23:32:39.723340 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:39.723346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:39.723431 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:39.723509 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:39.724774 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.725932 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.727858 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:39.728501 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:39.728660 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:39.730980 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:39.730987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:39.730992 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:39.731039 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:39.734733 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:39.735596 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:39.735604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:39.735610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:39.735770 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:39.736231 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:39.736236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:39.736240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:39.736674 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:39.736846 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:39.737361 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 23:32:39.737622 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:32:39.739588 19495 tablet_copy_service.cc:140] P 2724aeacf453402d98aecdbb20891190: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ed7ac368f3fe4686b6349f2599e4d93f ({username='oryx', principal='oryx/127.9.48.253@KRBTEST.COM'} at 127.0.0.1:32942)
I20260810 23:32:39.740007 19495 tablet_copy_service.cc:161] P 2724aeacf453402d98aecdbb20891190: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ed7ac368f3fe4686b6349f2599e4d93f at {username='oryx', principal='oryx/127.9.48.253@KRBTEST.COM'} at 127.0.0.1:32942: session id = ed7ac368f3fe4686b6349f2599e4d93f-00000000000000000000000000000000
I20260810 23:32:39.745083 19495 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 23:32:39.747943 20222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 23:32:39.755036 20222 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: tablet copy: Starting download of 0 data blocks...
I20260810 23:32:39.755463 20222 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: tablet copy: Starting download of 1 WAL segments...
I20260810 23:32:39.759311 20222 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 23:32:39.763887 19495 tablet_copy_service.cc:342] P 2724aeacf453402d98aecdbb20891190: Request end of tablet copy session ed7ac368f3fe4686b6349f2599e4d93f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.48.253@KRBTEST.COM'} at 127.0.0.1:32942
I20260810 23:32:39.764164 19495 tablet_copy_service.cc:433] P 2724aeacf453402d98aecdbb20891190: ending tablet copy session ed7ac368f3fe4686b6349f2599e4d93f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ed7ac368f3fe4686b6349f2599e4d93f
I20260810 23:32:39.776971 20222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:39.779995 20419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:39.781903 20420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:39.789585 20222 server_base.cc:1034] running on GCE node
W20260810 23:32:39.790220 20423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:39.790638 20222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:39.791324 20222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:39.792569 20222 hybrid_clock.cc:648] HybridClock initialized: now 1786404759792491 us; error 28 us; skew 500 ppm
I20260810 23:32:39.792806 20222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:39 dist-test-slave-04bb krb5kdc[19390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404759, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:39.795204 20222 init.cc:377] Logged in from keytab as oryx/127.9.48.253@KRBTEST.COM (short username oryx)
I20260810 23:32:39.801810 20222 webserver.cc:460] Webserver started at http://127.9.48.253:35485/ using document root <none> and password file <none>
I20260810 23:32:39.802171 20222 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:39.802264 20222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:39.805543 20222 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:32:39.807642 20435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:39.808135 20222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:32:39.808293 20222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/wal
uuid: "ed7ac368f3fe4686b6349f2599e4d93f"
format_stamp: "Formatted at 2026-08-10 23:32:39 on dist-test-slave-04bb"
server_key: "78594f70cb971cb8ef0c11c4f975a0ac"
server_key_iv: "c3a971a99e9c437bd4f022b979227b50"
server_key_version: "encryptionkey@0"
I20260810 23:32:39.808459 20222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:39.834038 20222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:39.838239 20222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:39.870673 20222 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.253:35727
I20260810 23:32:39.870743 20522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.253:35727 every 8 connection(s)
I20260810 23:32:39.871948 20222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-1/data/info.pb
I20260810 23:32:39.877756 20523 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:32:39.882318 20523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: Bootstrap starting.
I20260810 23:32:39.902891 20523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:32:39.903517 20523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: Bootstrap complete.
I20260810 23:32:39.904778 20523 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } }
I20260810 23:32:39.905156 20523 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ed7ac368f3fe4686b6349f2599e4d93f, State: Initialized, Role: LEARNER
I20260810 23:32:39.905495 20523 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } }
I20260810 23:32:39.906469 20529 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } } }
I20260810 23:32:39.906854 20523 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:39.906908 20529 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: This master's current role is: LEARNER
I20260810 23:32:39.916049 20545 catalog_manager.cc:1259] Loaded cluster ID: e341c7ce902047f2843c6a47b0eb866a
I20260810 23:32:39.916152 20545 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: loading cluster ID for follower catalog manager: success
I20260810 23:32:39.919827 20545 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: acquiring CA information for follower catalog manager: success
I20260810 23:32:39.922508 20545 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:32:39.984090 20213 heartbeater.cc:503] Master 127.9.48.254:40553 was elected leader, sending a full tablet report...
W20260810 23:32:40.113696 20400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:40.114040 20400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:40.114236 20400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:40.125327 20400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:40.125618 20400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:40.125808 20400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:40.125994 20400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:40.126168 20400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:40.130848 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.120674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:35727 (local address 127.0.0.1:37348)
0810 23:32:40.120949 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:40.120974 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:40.121145 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:40.122857 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:40.122863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:40.122893 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:40.123321 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.123334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.125561 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.125566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.127545 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.127559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.129662 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.129670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.130027 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.130101 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 23:32:40.130229 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 23:32:40.130405 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 23:32:40.131120 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.121083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.9.48.253:35727)
0810 23:32:40.122492 (+ 1409us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:40.122496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:40.122513 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:32:40.122578 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:40.122582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:40.122665 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:40.122814 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:40.123564 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.125327 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.127708 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.129214 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.129577 (+ 363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.130000 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:40.130273 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 23:32:40.130453 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1278,"thread_start_us":983,"threads_started":1}
I20260810 23:32:40.132746 20486 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260810 23:32:40.133615 20529 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2724aeacf453402d98aecdbb20891190. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } } }
I20260810 23:32:40.133993 20529 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: This master's current role is: LEARNER
I20260810 23:32:40.141412 20400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:40553,127.9.48.253:35727,127.9.48.252:34131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.252.keytab
--principal=oryx/127.9.48.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.252:34131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.48.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:40.143114 20400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:40.145061 20400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:40.161731 20555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:40.161654 20558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:40.162014 20560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:40.163566 20400 server_base.cc:1034] running on GCE node
I20260810 23:32:40.164448 20400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:40.166215 20400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:40.167568 20400 hybrid_clock.cc:648] HybridClock initialized: now 1786404760167445 us; error 49 us; skew 500 ppm
I20260810 23:32:40.168082 20400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:40 dist-test-slave-04bb krb5kdc[19390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404760, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:40.172770 20400 init.cc:377] Logged in from keytab as oryx/127.9.48.252@KRBTEST.COM (short username oryx)
I20260810 23:32:40.175025 20400 webserver.cc:460] Webserver started at http://127.9.48.252:36653/ using document root <none> and password file <none>
I20260810 23:32:40.175901 20400 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:40.176126 20400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:40.176589 20400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:40.180732 20400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/data/instance:
uuid: "1a4caf350fc747079975acc1aceb6a3c"
format_stamp: "Formatted at 2026-08-10 23:32:40 on dist-test-slave-04bb"
server_key: "bd6e2f498935abc9c6c83f747a2b42ce"
server_key_iv: "6ea9875a396baaccb33afbc96cc72c76"
server_key_version: "encryptionkey@0"
I20260810 23:32:40.181867 20400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/wal/instance:
uuid: "1a4caf350fc747079975acc1aceb6a3c"
format_stamp: "Formatted at 2026-08-10 23:32:40 on dist-test-slave-04bb"
server_key: "bd6e2f498935abc9c6c83f747a2b42ce"
server_key_iv: "6ea9875a396baaccb33afbc96cc72c76"
server_key_version: "encryptionkey@0"
I20260810 23:32:40.188971 20400 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260810 23:32:40.193112 20571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:40.194684 20400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:32:40.194977 20400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/wal
uuid: "1a4caf350fc747079975acc1aceb6a3c"
format_stamp: "Formatted at 2026-08-10 23:32:40 on dist-test-slave-04bb"
server_key: "bd6e2f498935abc9c6c83f747a2b42ce"
server_key_iv: "6ea9875a396baaccb33afbc96cc72c76"
server_key_version: "encryptionkey@0"
I20260810 23:32:40.195297 20400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:40.217245 20400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:40.223080 20400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:40.223469 20400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:40.261108 20400 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.252:34131
I20260810 23:32:40.261317 20647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.252:34131 every 8 connection(s)
I20260810 23:32:40.262995 20400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/data/info.pb
I20260810 23:32:40.265244 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 20400
I20260810 23:32:40.269973 20648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:40.280016 20648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:32:40 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.48.252@KRBTEST.COM
Aug 10 23:32:40 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404760, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.252@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
I20260810 23:32:40.298657 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.268092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58550 (local address 127.9.48.252:34131)
0810 23:32:40.269399 (+ 1307us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:40.269411 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:40.269452 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:32:40.271975 (+ 2523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:40.271997 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:40.272404 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:40.272943 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:40.275328 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.277312 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.279071 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.280001 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.280202 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.288022 (+ 7820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:40.288031 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:40.288044 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:40.288080 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:40.292699 (+ 4619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.293554 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.293562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.293567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.293846 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.294525 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.294528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.294531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.295148 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:40.295499 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:40.296155 (+ 656us) server_negotiation.cc:299] Negotiation successful
0810 23:32:40.296873 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1072,"thread_start_us":762,"threads_started":1}
I20260810 23:32:40.306304 20653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.283608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32948 (local address 127.9.48.254:40553)
0810 23:32:40.284824 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:40.284843 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:40.285709 (+ 866us) server_negotiation.cc:407] Connection header received
0810 23:32:40.285966 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:40.285970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:40.286036 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:40.286174 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:40.288384 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.289616 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.291997 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.292649 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.292783 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.298202 (+ 5419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:40.298207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:40.298211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:40.298242 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:40.303149 (+ 4907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.304186 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.304190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.304192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.304390 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.304793 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.304797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.304799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.305124 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:40.305230 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:40.305761 (+ 531us) server_negotiation.cc:299] Negotiation successful
0810 23:32:40.305976 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":978,"thread_start_us":734,"threads_started":1}
I20260810 23:32:40.306399 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.283945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:32948)
0810 23:32:40.285554 (+ 1609us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:40.285590 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:40.285818 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:40.286226 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:40.286233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:40.287487 (+ 1254us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:40.288218 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.288230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.289868 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.289872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.291845 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.291858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.292957 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.292967 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.293177 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.294056 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:40.294136 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:40.297967 (+ 3831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:40.303275 (+ 5308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.303283 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.303287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.304071 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.304499 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.304502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.304505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.304696 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.305253 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:40.305258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:40.305436 (+ 178us) client_negotiation.cc:769] Sending connection context
0810 23:32:40.305639 (+ 203us) client_negotiation.cc:240] Negotiation successful
0810 23:32:40.305921 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1367,"thread_start_us":1152,"threads_started":1}
I20260810 23:32:40.308095 20648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:32:40 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404760, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.252@KRBTEST.COM for oryx/127.9.48.253@KRBTEST.COM
I20260810 23:32:40.325105 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.309190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:35727 (local address 127.0.0.1:37354)
0810 23:32:40.309436 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:40.309456 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:40.309590 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:40.309943 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:40.309946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:40.310291 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:40.310468 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.310477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.311953 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.311957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.313397 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.313409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.314461 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.314465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.314682 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.315478 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:40.315495 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:40.318186 (+ 2691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:40.322352 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.322356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.322358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.322875 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.323360 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.323363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.323365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.323522 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.324053 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:40.324057 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:40.324172 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 23:32:40.324298 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 23:32:40.324685 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 23:32:40.325265 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.309418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.9.48.253:35727)
0810 23:32:40.309598 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:40.309603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:40.309643 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:32:40.309719 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:40.309724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:40.309788 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:40.309917 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:40.310663 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.311789 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.313539 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.314266 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.314370 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.318424 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:40.318430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:40.318433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:40.318466 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:40.322225 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.322986 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.322991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.322993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.323257 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.323616 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.323619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.323621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.323949 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:40.324066 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:40.324452 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 23:32:40.324737 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:32:40.326388 20648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:32:40 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404760, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.252@KRBTEST.COM for oryx/127.9.48.252@KRBTEST.COM
I20260810 23:32:40.338197 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.327181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:34131 (local address 127.0.0.1:58566)
0810 23:32:40.327418 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:40.327437 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:40.327539 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:40.327876 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:40.327880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:40.328247 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:40.328377 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.328385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.329341 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.329348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.330157 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.330168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.330881 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.330887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.331070 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.331530 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:40.331546 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:40.333499 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:40.336109 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.336113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.336115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.336452 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.336846 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.336850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.336852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.336995 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.337434 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:40.337437 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:40.337560 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 23:32:40.337683 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 23:32:40.337854 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 23:32:40.338301 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.327304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.9.48.252:34131)
0810 23:32:40.327452 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:40.327460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:40.327480 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:32:40.327687 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:40.327696 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:40.327747 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:40.327858 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:40.328516 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.329216 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.330298 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.330720 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.330841 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.333662 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:40.333667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:40.333671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:40.333699 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:40.335979 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.336608 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.336612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.336614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.336741 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.337091 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.337094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.337096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.337336 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:40.337443 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:40.337799 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 23:32:40.337973 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:32:40.343137 20648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: Bootstrap starting.
I20260810 23:32:40.346375 20648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:40.347404 20648 log.cc:824] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:40.350032 20648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: No bootstrap required, opened a new log
I20260810 23:32:40.357585 20648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } }
I20260810 23:32:40.357883 20648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:40.358017 20648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a4caf350fc747079975acc1aceb6a3c, State: Initialized, Role: FOLLOWER
I20260810 23:32:40.358667 20648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } }
I20260810 23:32:40.359572 20656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } } }
I20260810 23:32:40.359949 20656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:40.360590 20648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:32:40.370674 20674 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:32:40.370824 20674 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:32:40.373939 20400 master_runner.cc:428] Detected that this master 1a4caf350fc747079975acc1aceb6a3c is joining an existing cluster
I20260810 23:32:40.374039 20400 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.48.252:34131
I20260810 23:32:40.404686 20653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.378977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32950 (local address 127.9.48.254:40553)
0810 23:32:40.379342 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:40.379350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:40.385901 (+ 6551us) server_negotiation.cc:407] Connection header received
0810 23:32:40.386558 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:40.386567 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:40.386679 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:40.386768 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:40.389042 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.390393 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.392680 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.393366 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.393461 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.396330 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:40.396336 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:40.396341 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:40.396378 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:40.400136 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.402059 (+ 1923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.402070 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.402074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.402236 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.402806 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.402812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.402815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.403181 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:40.403321 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:40.404064 (+ 743us) server_negotiation.cc:299] Negotiation successful
0810 23:32:40.404302 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260810 23:32:40.404733 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.379143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.9.48.252:34131)
0810 23:32:40.379325 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:40.379331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:40.387735 (+ 8404us) server_negotiation.cc:407] Connection header received
0810 23:32:40.388342 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:40.388347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:40.388431 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:40.388501 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:40.390012 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.391474 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.394206 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.395014 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.395121 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.398006 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:40.398012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:40.398017 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:40.398051 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:40.401491 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.402506 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.402511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.402515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.402654 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.403046 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.403050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.403054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.403399 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:40.403560 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:40.404076 (+ 516us) server_negotiation.cc:299] Negotiation successful
0810 23:32:40.404335 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:32:40.405534 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 19557
I20260810 23:32:40.406329 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.378332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:34131 (local address 127.0.0.1:58582)
0810 23:32:40.387622 (+ 9290us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:40.387643 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:40.388059 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:40.388728 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:40.388735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:40.389480 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:40.389841 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.389853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.392053 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.392059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.394015 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.394037 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.395465 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.395474 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.395667 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.396335 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:40.396372 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:40.397825 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:40.401857 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.401863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.401867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.402377 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.402784 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.402788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.402792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.402931 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.403643 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:40.403650 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:40.403803 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 23:32:40.404298 (+ 495us) client_negotiation.cc:240] Negotiation successful
0810 23:32:40.405831 (+ 1533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9125,"mutex_wait_us":29,"thread_start_us":12377,"threads_started":1}
I20260810 23:32:40.407749 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.378147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.9.48.253:35727)
0810 23:32:40.378334 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:40.378340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:40.390705 (+ 12365us) server_negotiation.cc:407] Connection header received
0810 23:32:40.391198 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:40.391203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:40.391300 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:40.391370 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:40.392879 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.393924 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.396259 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.396907 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.397014 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.399946 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:40.399955 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:40.399962 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:40.400005 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:40.403676 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.405156 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.405161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.405164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.405336 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.405838 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.405844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.405847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.406221 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:40.406363 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:40.406953 (+ 590us) server_negotiation.cc:299] Negotiation successful
0810 23:32:40.407270 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:32:40.409391 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.378296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:32950)
0810 23:32:40.385754 (+ 7458us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:40.385780 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:40.386346 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:40.387049 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:40.387056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:40.388032 (+ 976us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:40.388792 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.388816 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.390619 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.390625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.392468 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.392482 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.393801 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.393808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.394076 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.394984 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:40.395009 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:40.396135 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:40.400287 (+ 4152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.400293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.400297 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.401922 (+ 1625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.402397 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.402405 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.402410 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.402652 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.403409 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:40.403415 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:40.403637 (+ 222us) client_negotiation.cc:769] Sending connection context
0810 23:32:40.404881 (+ 1244us) client_negotiation.cc:240] Negotiation successful
0810 23:32:40.408846 (+ 3965us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7265,"mutex_wait_us":3600,"thread_start_us":9452,"threads_started":1}
I20260810 23:32:40.409125 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.378795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:35727 (local address 127.0.0.1:37360)
0810 23:32:40.390606 (+ 11811us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:40.390623 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:40.391033 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:40.391595 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:40.391601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:40.392320 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:40.392719 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.392731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.394164 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.394171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.396068 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.396082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.397376 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.397382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.397572 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.398495 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:40.398521 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:40.399739 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:40.403915 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.403923 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.403930 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.405011 (+ 1081us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.405502 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.405507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.405513 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.405693 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.406450 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:40.406454 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:40.406650 (+ 196us) client_negotiation.cc:769] Sending connection context
0810 23:32:40.407945 (+ 1295us) client_negotiation.cc:240] Negotiation successful
0810 23:32:40.408399 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11661,"thread_start_us":7116,"threads_started":1}
Aug 10 23:32:40 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.48.193@KRBTEST.COM
Aug 10 23:32:40 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404757, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.254@KRBTEST.COM for oryx/127.9.48.252@KRBTEST.COM
I20260810 23:32:40.430249 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.406924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.9.48.193:46437)
0810 23:32:40.410129 (+ 3205us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:40.410152 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:40.410197 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:32:40.410302 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:40.410312 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:40.410610 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:40.410802 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:40.412613 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.414226 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.415705 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.416593 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.417342 (+ 749us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.422709 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:40.422723 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:40.422738 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:40.422770 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:40.426749 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.427423 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.427431 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.427435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.427572 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.428125 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.428130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.428134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.428484 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:40.428774 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:40.429503 (+ 729us) server_negotiation.cc:299] Negotiation successful
0810 23:32:40.429788 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2731,"thread_start_us":2362,"threads_started":1}
I20260810 23:32:40.431764 19752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:40.435675 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.416799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:34131 (local address 127.0.0.1:58588)
0810 23:32:40.417168 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:40.417187 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:40.417757 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:40.418344 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:40.418349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:40.418821 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:40.419050 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.419065 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.420581 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.420588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.422033 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.422054 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.423107 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.423114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.423308 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.423983 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:40.424011 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:40.426681 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:40.432224 (+ 5543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.432229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.432233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.432791 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.433423 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.433428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.433431 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.433720 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.434614 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:40.434619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:40.434794 (+ 175us) client_negotiation.cc:769] Sending connection context
0810 23:32:40.434954 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 23:32:40.435252 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260810 23:32:40.436616 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.417604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.9.48.252:34131)
0810 23:32:40.417757 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:40.417763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:40.417782 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:32:40.417904 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:40.417912 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:40.417999 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:40.418105 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:40.419242 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.420432 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.422237 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.422932 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.423068 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.426908 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:40.426915 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:40.426921 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:40.426960 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:40.432042 (+ 5082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.432968 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.432980 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.432983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.433248 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.433948 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.433979 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.433985 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.434377 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:40.434544 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:40.435996 (+ 1452us) server_negotiation.cc:299] Negotiation successful
0810 23:32:40.436248 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:32:40.438078 19466 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.48.252:34131: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c"
member_type: NON_VOTER
last_known_addr {
host: "127.9.48.252"
port: 34131
}
attrs {
promote: true
}
}
dest_uuid: "2724aeacf453402d98aecdbb20891190"
cas_config_opid_index: 5
I20260810 23:32:40.438951 19466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } }
I20260810 23:32:40.440704 20396 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1a4caf350fc747079975acc1aceb6a3c (127.9.48.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } } }
I20260810 23:32:40.445389 20486 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 LEARNER]: Refusing update from remote peer 2724aeacf453402d98aecdbb20891190: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 23:32:40.446182 20609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:40.450649 20609 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 FOLLOWER]: Refusing update from remote peer 2724aeacf453402d98aecdbb20891190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 23:32:40.457291 20693 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 23:32:40.457940 20693 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:40.464445 20396 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.48.252:34131
I20260810 23:32:40.465051 20400 master.cc:537] Master@127.9.48.252:34131 shutting down...
I20260810 23:32:40.465157 20693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } } }
I20260810 23:32:40.465534 20693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:40.472786 20486 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1a4caf350fc747079975acc1aceb6a3c (127.9.48.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } } }
I20260810 23:32:40.474067 20609 raft_consensus.cc:858] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } }
W20260810 23:32:40.474255 20609 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } }
I20260810 23:32:40.479609 20529 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } } }
I20260810 23:32:40.479955 20529 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: This master's current role is: LEARNER
I20260810 23:32:40.480557 20396 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190: attempting to promote NON_VOTER ed7ac368f3fe4686b6349f2599e4d93f to VOTER
I20260810 23:32:40.481478 20396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } }
I20260810 23:32:40.484028 20486 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 LEARNER]: Refusing update from remote peer 2724aeacf453402d98aecdbb20891190: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 23:32:40.484669 20693 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 23:32:40.487051 20694 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ed7ac368f3fe4686b6349f2599e4d93f (127.9.48.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } } }
I20260810 23:32:40.487673 20486 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ed7ac368f3fe4686b6349f2599e4d93f (127.9.48.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } } }
I20260810 23:32:40.491282 20694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } } }
I20260810 23:32:40.491655 20694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:40.491677 20529 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } } }
I20260810 23:32:40.491922 20529 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:40.492094 20396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } } }
I20260810 23:32:40.492326 20396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: This master's current role is: LEADER
W20260810 23:32:40.493497 20579 tablet.cc:2369] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:32:40.506824 19433 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 -> Peer 1a4caf350fc747079975acc1aceb6a3c (127.9.48.252:34131): Couldn't send request to peer 1a4caf350fc747079975acc1aceb6a3c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:40.520098 20609 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ed7ac368f3fe4686b6349f2599e4d93f (127.9.48.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } } }
I20260810 23:32:40.525472 20656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2724aeacf453402d98aecdbb20891190. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } } }
I20260810 23:32:40.525730 20656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 23:32:40.525936 20656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } } }
I20260810 23:32:40.526120 20656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 23:32:40.526448 20702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: NON_VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: true } } }
I20260810 23:32:40.527346 20702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 23:32:40.529965 20400 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260810 23:32:40.530452 20400 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260810 23:32:40.530637 20400 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: stopping tablet replica
I20260810 23:32:40.537954 20400 master.cc:552] Master@127.9.48.252:34131 shutdown complete.
I20260810 23:32:40.538066 20400 master_runner.cc:305] Clearing existing system tablet
I20260810 23:32:40.538431 20400 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:40.538559 20400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:40.540840 20400 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:32:40.542344 20707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:40.542627 20400 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:40.542915 20400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260810 23:32:40.543015 20400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/wal
uuid: "1a4caf350fc747079975acc1aceb6a3c"
format_stamp: "Formatted at 2026-08-10 23:32:40 on dist-test-slave-04bb"
server_key: "bd6e2f498935abc9c6c83f747a2b42ce"
server_key_iv: "6ea9875a396baaccb33afbc96cc72c76"
server_key_version: "encryptionkey@0"
I20260810 23:32:40.546242 20400 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:40.551031 20400 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 23:32:40.551210 20400 log.cc:1196] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 23:32:40.551693 20400 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: Deleting consensus metadata
I20260810 23:32:40.551964 20400 master_runner.cc:315] Copying system tablet from 127.9.48.254:40553
I20260810 23:32:40.555013 20400 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: tablet copy: Beginning tablet copy session from remote peer at address 127.9.48.254:40553
I20260810 23:32:40.568863 20653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.555785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.9.48.254:40553)
0810 23:32:40.556084 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:40.556093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:40.556749 (+ 656us) server_negotiation.cc:407] Connection header received
0810 23:32:40.556972 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:40.556978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:40.557089 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:40.557208 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:40.558019 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.558759 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.560027 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.560669 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.560800 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.563016 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:40.563022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:40.563027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:40.563107 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:40.566101 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.566727 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.566734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.566738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.566874 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:40.567264 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:40.567270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:40.567273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:40.567545 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:40.567705 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:40.568110 (+ 405us) server_negotiation.cc:299] Negotiation successful
0810 23:32:40.568330 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260810 23:32:40.568889 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.555741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:32962)
0810 23:32:40.556624 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:40.556656 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:40.556814 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:40.557232 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:40.557237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:40.557679 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:40.557867 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.557877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.558911 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.558916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.559882 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.559895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.560824 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.560830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.561048 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.561796 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:40.561825 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:40.562799 (+ 974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:40.566251 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.566257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.566261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.566596 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.566988 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:40.566993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:40.566997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:40.567153 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:40.567677 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:40.567682 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:40.567873 (+ 191us) client_negotiation.cc:769] Sending connection context
0810 23:32:40.568020 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 23:32:40.568335 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":501,"threads_started":1}
I20260810 23:32:40.569788 19495 tablet_copy_service.cc:140] P 2724aeacf453402d98aecdbb20891190: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1a4caf350fc747079975acc1aceb6a3c ({username='oryx', principal='oryx/127.9.48.252@KRBTEST.COM'} at 127.0.0.1:32962)
I20260810 23:32:40.569969 19495 tablet_copy_service.cc:161] P 2724aeacf453402d98aecdbb20891190: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1a4caf350fc747079975acc1aceb6a3c at {username='oryx', principal='oryx/127.9.48.252@KRBTEST.COM'} at 127.0.0.1:32962: session id = 1a4caf350fc747079975acc1aceb6a3c-00000000000000000000000000000000
I20260810 23:32:40.572371 19495 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 23:32:40.574070 20400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 23:32:40.579265 20400 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: tablet copy: Starting download of 0 data blocks...
I20260810 23:32:40.579595 20400 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: tablet copy: Starting download of 1 WAL segments...
I20260810 23:32:40.581830 20400 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 23:32:40.585009 19495 tablet_copy_service.cc:342] P 2724aeacf453402d98aecdbb20891190: Request end of tablet copy session 1a4caf350fc747079975acc1aceb6a3c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.48.252@KRBTEST.COM'} at 127.0.0.1:32962
I20260810 23:32:40.585157 19495 tablet_copy_service.cc:433] P 2724aeacf453402d98aecdbb20891190: ending tablet copy session 1a4caf350fc747079975acc1aceb6a3c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1a4caf350fc747079975acc1aceb6a3c
I20260810 23:32:40.595960 20400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:40.598623 20715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:40.604318 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 19557
W20260810 23:32:40.612871 20716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:40.621027 20400 server_base.cc:1034] running on GCE node
W20260810 23:32:40.622213 20720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:40.622700 20400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:40.623477 20400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:40.624627 20400 hybrid_clock.cc:648] HybridClock initialized: now 1786404760624586 us; error 34 us; skew 500 ppm
I20260810 23:32:40.624897 20400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:40 dist-test-slave-04bb krb5kdc[19390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404760, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:40.626911 20400 init.cc:377] Logged in from keytab as oryx/127.9.48.252@KRBTEST.COM (short username oryx)
I20260810 23:32:40.633538 20400 webserver.cc:460] Webserver started at http://127.9.48.252:42201/ using document root <none> and password file <none>
I20260810 23:32:40.633843 20400 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:40.633960 20400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:40.636395 20400 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:32:40.637715 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:46437
--local_ip_for_outbound_sockets=127.9.48.193
--tserver_master_addrs=127.9.48.254:40553
--webserver_port=35211
--webserver_interface=127.9.48.193
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.193.keytab
--principal=oryx/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.48.254:40553,127.9.48.253:35727
--tserver_master_addrs=127.9.48.254:40553,127.9.48.253:35727,127.9.48.252:34131 with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:32:40.638578 20730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:40.639048 20400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:32:40.639225 20400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/wal
uuid: "1a4caf350fc747079975acc1aceb6a3c"
format_stamp: "Formatted at 2026-08-10 23:32:40 on dist-test-slave-04bb"
server_key: "bd6e2f498935abc9c6c83f747a2b42ce"
server_key_iv: "6ea9875a396baaccb33afbc96cc72c76"
server_key_version: "encryptionkey@0"
I20260810 23:32:40.639477 20400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:40.671262 20400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:40.675716 20400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:40.703771 20400 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.252:34131
I20260810 23:32:40.703828 20822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.252:34131 every 8 connection(s)
I20260810 23:32:40.705114 20400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/master-2/data/info.pb
I20260810 23:32:40.711184 20823 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:32:40.715030 20823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: Bootstrap starting.
I20260810 23:32:40.729611 20823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:32:40.730134 20823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: Bootstrap complete.
I20260810 23:32:40.731212 20823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } }
I20260810 23:32:40.731515 20823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1a4caf350fc747079975acc1aceb6a3c, State: Initialized, Role: LEARNER
I20260810 23:32:40.731773 20823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } }
I20260810 23:32:40.733007 20823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:40.732724 20827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } } }
I20260810 23:32:40.733328 20827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: This master's current role is: LEARNER
I20260810 23:32:40.741701 20843 catalog_manager.cc:1259] Loaded cluster ID: e341c7ce902047f2843c6a47b0eb866a
I20260810 23:32:40.741801 20843 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: loading cluster ID for follower catalog manager: success
I20260810 23:32:40.745391 20843 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: acquiring CA information for follower catalog manager: success
I20260810 23:32:40.748234 20843 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:32:40.945730 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.938550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:34131 (local address 127.0.0.1:58592)
0810 23:32:40.939105 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:40.939133 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:40.939300 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:40.939892 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:40.939896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:40.939919 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:40.940114 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.940122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.942029 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.942034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.943276 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:40.943289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.945052 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.945056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:40.945177 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.945203 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:32:40.945268 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 23:32:40.945370 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":264,"threads_started":1}
I20260810 23:32:40.945955 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:40.938740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.9.48.252:34131)
0810 23:32:40.939479 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:40.939486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:40.939510 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:32:40.939597 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:40.939605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:40.939719 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:40.939878 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:40.940278 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.941871 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.943456 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:40.944888 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:40.945042 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:40.945329 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:40.945396 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 23:32:40.945539 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":420,"threads_started":1}
I20260810 23:32:40.946797 20783 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 23:32:40.947279 20827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2724aeacf453402d98aecdbb20891190. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } } }
I20260810 23:32:40.947584 20827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: This master's current role is: LEARNER
W20260810 23:32:41.012228 20740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:41.012703 20740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:41.012873 20740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:41.020116 20740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:41.020272 20740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:41.020443 20740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:32:41.030246 20740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.193.keytab
--principal=oryx/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:46437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=35211
--tserver_master_addrs=127.9.48.254:40553,127.9.48.253:35727,127.9.48.252:34131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:41.031581 20740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:41.033089 20740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:41.046959 20859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:41.046919 20860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:41.047155 20863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:41.047717 20740 server_base.cc:1034] running on GCE node
I20260810 23:32:41.048319 20740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:41.049291 20740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:41.050496 20740 hybrid_clock.cc:648] HybridClock initialized: now 1786404761050454 us; error 41 us; skew 500 ppm
I20260810 23:32:41.050819 20740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:41 dist-test-slave-04bb krb5kdc[19390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404761, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:41.054129 20740 init.cc:377] Logged in from keytab as oryx/127.9.48.193@KRBTEST.COM (short username oryx)
I20260810 23:32:41.055540 20740 webserver.cc:460] Webserver started at http://127.9.48.193:35211/ using document root <none> and password file <none>
I20260810 23:32:41.056178 20740 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:41.056293 20740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:41.061623 20740 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:32:41.064363 20873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:41.065706 20740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:32:41.065838 20740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "c94048588350439e91217f75d0058224"
format_stamp: "Formatted at 2026-08-10 23:32:37 on dist-test-slave-04bb"
server_key: "96c7abf6a75771e9fc418a4d67c53070"
server_key_iv: "cf9153dbcae858fb1080cc15600e556b"
server_key_version: "encryptionkey@0"
I20260810 23:32:41.066759 20740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:41.093268 20740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:41.098083 20740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:41.098452 20740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:41.099411 20740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:41.101068 20740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:41.101177 20740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:41.101276 20740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:41.101339 20740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:41.142525 20740 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:46437
I20260810 23:32:41.142625 21048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:46437 every 8 connection(s)
I20260810 23:32:41.144084 20740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:32:41.154435 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 20740
I20260810 23:32:41.154851 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 19768
Aug 10 23:32:41 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.48.194@KRBTEST.COM
Aug 10 23:32:41 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404761, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.252@KRBTEST.COM
Aug 10 23:32:41 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404761, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.253@KRBTEST.COM
Aug 10 23:32:41 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404761, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
I20260810 23:32:41.187541 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:41.156883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44726 (local address 127.9.48.194:43199)
0810 23:32:41.162416 (+ 5533us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:41.162429 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:41.163068 (+ 639us) server_negotiation.cc:407] Connection header received
0810 23:32:41.163391 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:41.163399 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:41.163664 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:41.163841 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:41.165866 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.167446 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.169442 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.170349 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.171187 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:41.176985 (+ 5798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:41.177002 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:41.177021 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:41.177066 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:41.183142 (+ 6076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.184037 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.184046 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.184049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.184214 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.184711 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.184719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.184724 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.185175 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:41.185509 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:41.186802 (+ 1293us) server_negotiation.cc:299] Negotiation successful
0810 23:32:41.187073 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5221,"thread_start_us":5004,"threads_started":1}
I20260810 23:32:41.188079 21057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:41.147245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:49023 (local address 127.9.48.254:40553)
0810 23:32:41.159714 (+ 12469us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:41.159726 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:41.163797 (+ 4071us) server_negotiation.cc:407] Connection header received
0810 23:32:41.164722 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:41.164728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:41.164818 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:41.164934 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:41.167913 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.169194 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.172448 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.173200 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.173297 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:41.179987 (+ 6690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:41.179995 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:41.180000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:41.180044 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:41.183736 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.185061 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.185067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.185070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.185224 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.185795 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.185800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.185804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.186147 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:41.186276 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:41.187417 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0810 23:32:41.187655 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12264,"thread_start_us":12478,"threads_started":1}
I20260810 23:32:41.188165 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:41.148179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:59123 (local address 127.9.48.253:35727)
0810 23:32:41.160252 (+ 12073us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:41.160258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:41.161249 (+ 991us) server_negotiation.cc:407] Connection header received
0810 23:32:41.164582 (+ 3333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:41.164591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:41.164697 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:41.164823 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:41.167898 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.169031 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.172406 (+ 3375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.173253 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.173374 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:41.179200 (+ 5826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:41.179209 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:41.179214 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:41.179260 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:41.183585 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.185071 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.185078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.185082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.185247 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.185814 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.185820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.185824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.186182 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:41.186315 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:41.187453 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0810 23:32:41.187728 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11941,"thread_start_us":11830,"threads_started":1}
I20260810 23:32:41.188802 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:41.147394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:47031 (local address 127.9.48.252:34131)
0810 23:32:41.147653 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:41.147663 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:41.161248 (+ 13585us) server_negotiation.cc:407] Connection header received
0810 23:32:41.164762 (+ 3514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:41.164768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:41.164854 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:41.164935 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:41.167914 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.169194 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.172399 (+ 3205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.173183 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.173297 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:41.179032 (+ 5735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:41.179046 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:41.179053 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:41.179111 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:41.183896 (+ 4785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.185071 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.185077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.185082 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.185255 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.185814 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.185820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.185824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.186193 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:41.186314 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:41.187451 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0810 23:32:41.188004 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 23:32:41.189083 19985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:41.190274 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:41.148911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.193:49023)
0810 23:32:41.163683 (+ 14772us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:41.163702 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:41.164484 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:41.165304 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:41.165317 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:41.166405 (+ 1088us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:41.167687 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:41.167698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.169412 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.169418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:41.172286 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:41.172301 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.173698 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.173707 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:41.174274 (+ 567us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:41.175710 (+ 1436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:41.175736 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:41.179758 (+ 4022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:41.183903 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:41.183910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:41.183917 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:41.184915 (+ 998us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:41.185375 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:41.185380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:41.185384 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:41.185640 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:41.186387 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:41.186396 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:41.186944 (+ 548us) client_negotiation.cc:769] Sending connection context
0810 23:32:41.188335 (+ 1391us) client_negotiation.cc:240] Negotiation successful
0810 23:32:41.188895 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14624,"mutex_wait_us":730,"spinlock_wait_cycles":33024,"thread_start_us":14454,"threads_started":1}
I20260810 23:32:41.191040 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:41.148022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:34131 (local address 127.9.48.193:47031)
0810 23:32:41.161072 (+ 13050us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:41.161110 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:41.164210 (+ 3100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:41.165304 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:41.165317 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:41.166405 (+ 1088us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:41.167659 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:41.167690 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.169422 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.169426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:41.172198 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:41.172214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.173711 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.173719 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:41.174219 (+ 500us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:41.175316 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:41.175340 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:41.178723 (+ 3383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:41.184073 (+ 5350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:41.184079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:41.184083 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:41.184915 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:41.185417 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:41.185423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:41.185426 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:41.185663 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:41.186420 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:41.186425 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:41.186954 (+ 529us) client_negotiation.cc:769] Sending connection context
0810 23:32:41.189377 (+ 2423us) client_negotiation.cc:240] Negotiation successful
0810 23:32:41.189814 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12924,"mutex_wait_us":614,"spinlock_wait_cycles":158976,"thread_start_us":13394,"threads_started":1}
I20260810 23:32:41.192051 21052 heartbeater.cc:347] Connected to a master server at 127.9.48.254:40553
I20260810 23:32:41.192233 21050 heartbeater.cc:347] Connected to a master server at 127.9.48.252:34131
I20260810 23:32:41.192431 21050 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:41.192435 21052 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:41.193553 21052 heartbeater.cc:511] Master 127.9.48.254:40553 requested a full tablet report, sending...
I20260810 23:32:41.193660 21050 heartbeater.cc:511] Master 127.9.48.252:34131 requested a full tablet report, sending...
I20260810 23:32:41.195576 19465 ts_manager.cc:194] Re-registered known tserver with Master: c94048588350439e91217f75d0058224 (127.9.48.193:46437)
I20260810 23:32:41.196138 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:41.147962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:35727 (local address 127.9.48.193:59123)
0810 23:32:41.161037 (+ 13075us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:41.161107 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:41.164210 (+ 3103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:41.165309 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:41.165317 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:41.166409 (+ 1092us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:41.167681 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:41.167698 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.169247 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.169253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:41.172221 (+ 2968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:41.172243 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.173759 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.173768 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:41.174216 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:41.175154 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:41.175191 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:41.178980 (+ 3789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:41.183862 (+ 4882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:41.183872 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:41.183888 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:41.184915 (+ 1027us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:41.185417 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:41.185423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:41.185427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:41.185669 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:41.186420 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:41.186425 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:41.186954 (+ 529us) client_negotiation.cc:769] Sending connection context
0810 23:32:41.188447 (+ 1493us) client_negotiation.cc:240] Negotiation successful
0810 23:32:41.195834 (+ 7387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12607,"mutex_wait_us":7589,"spinlock_wait_cycles":36352,"thread_start_us":12349,"threads_started":1}
I20260810 23:32:41.196274 20767 ts_manager.cc:194] Registered new tserver with Master: c94048588350439e91217f75d0058224 (127.9.48.193:46437)
I20260810 23:32:41.196859 19465 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:49023
I20260810 23:32:41.197391 21054 heartbeater.cc:347] Connected to a master server at 127.9.48.253:35727
I20260810 23:32:41.197579 21054 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:41.198279 21054 heartbeater.cc:511] Master 127.9.48.253:35727 requested a full tablet report, sending...
I20260810 23:32:41.200266 20464 ts_manager.cc:194] Registered new tserver with Master: c94048588350439e91217f75d0058224 (127.9.48.193:46437)
I20260810 23:32:41.296018 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 19768
I20260810 23:32:41.327908 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:43199
--local_ip_for_outbound_sockets=127.9.48.194
--tserver_master_addrs=127.9.48.254:40553
--webserver_port=40773
--webserver_interface=127.9.48.194
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.194.keytab
--principal=oryx/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.48.254:40553,127.9.48.253:35727
--tserver_master_addrs=127.9.48.254:40553,127.9.48.253:35727,127.9.48.252:34131 with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:32:41.434778 20396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: NON_VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: true } } }
I20260810 23:32:41.435163 20396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:41.436517 21075 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190: attempting to promote NON_VOTER 1a4caf350fc747079975acc1aceb6a3c to VOTER
I20260810 23:32:41.437347 21075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } }
I20260810 23:32:41.439447 20783 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 LEARNER]: Refusing update from remote peer 2724aeacf453402d98aecdbb20891190: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 23:32:41.439576 20486 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 FOLLOWER]: Refusing update from remote peer 2724aeacf453402d98aecdbb20891190: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 23:32:41.440146 21075 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 23:32:41.440591 20396 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 23:32:41.443130 20396 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1a4caf350fc747079975acc1aceb6a3c (127.9.48.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } } }
I20260810 23:32:41.443984 20486 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1a4caf350fc747079975acc1aceb6a3c (127.9.48.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } } }
I20260810 23:32:41.444376 20783 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1a4caf350fc747079975acc1aceb6a3c (127.9.48.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } } }
I20260810 23:32:41.447480 20396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } } }
I20260810 23:32:41.447716 20396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:41.448000 20396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } } }
I20260810 23:32:41.448230 20396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:41.448146 21072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } } }
I20260810 23:32:41.448355 21072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:41.452980 21083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } } }
I20260810 23:32:41.453270 21083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:41.453822 21080 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2724aeacf453402d98aecdbb20891190" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } } }
I20260810 23:32:41.454201 21080 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:32:41.675182 21070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:41.675493 21070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:41.675640 21070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:41.683138 21070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:41.683279 21070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:41.683451 21070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:32:41.693241 21070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.194.keytab
--principal=oryx/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:43199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=40773
--tserver_master_addrs=127.9.48.254:40553,127.9.48.253:35727,127.9.48.252:34131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:41.694512 21070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:41.695838 21070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:41.710304 21093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:41.710517 21090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:41.710953 21091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:41.712069 21070 server_base.cc:1034] running on GCE node
I20260810 23:32:41.712792 21070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:41.713760 21070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:41.714987 21070 hybrid_clock.cc:648] HybridClock initialized: now 1786404761714938 us; error 42 us; skew 500 ppm
I20260810 23:32:41.715310 21070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:41 dist-test-slave-04bb krb5kdc[19390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404761, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:41.718576 21070 init.cc:377] Logged in from keytab as oryx/127.9.48.194@KRBTEST.COM (short username oryx)
I20260810 23:32:41.720047 21070 webserver.cc:460] Webserver started at http://127.9.48.194:40773/ using document root <none> and password file <none>
I20260810 23:32:41.720827 21070 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:41.720993 21070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:41.726315 21070 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 23:32:41.729092 21101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:41.730311 21070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:32:41.730500 21070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "2d7651a6bfd3488d88b587ed9db9ee90"
format_stamp: "Formatted at 2026-08-10 23:32:38 on dist-test-slave-04bb"
server_key: "8194dbf54e1b18e260739d6f27a02241"
server_key_iv: "bfd4fd3c924475c8d80d2617126a975f"
server_key_version: "encryptionkey@0"
I20260810 23:32:41.731464 21070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:41.754352 21070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:41.758630 21070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:41.758922 21070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:41.759868 21070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:41.761303 21070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:41.761436 21070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:41.761559 21070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:41.761655 21070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:41.795421 21070 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:43199
I20260810 23:32:41.795507 21269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:43199 every 8 connection(s)
I20260810 23:32:41.796960 21070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:32:41.803570 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 21070
I20260810 23:32:41.803920 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 20004
Aug 10 23:32:41 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404761, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.252@KRBTEST.COM
I20260810 23:32:41.838073 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:41.800460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:34131 (local address 127.9.48.194:33795)
0810 23:32:41.814272 (+ 13812us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:41.814320 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:41.816120 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:41.817551 (+ 1431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:41.817560 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:41.818114 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:41.818923 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:41.818961 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.820618 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.820624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:41.822359 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:41.822372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.823350 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.823360 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:41.823632 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:41.824510 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:41.824592 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:41.827646 (+ 3054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:41.832477 (+ 4831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:41.832490 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:41.832509 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:41.833227 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:41.833750 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:41.833757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:41.833762 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:41.834036 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:41.834830 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:41.834839 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:41.835288 (+ 449us) client_negotiation.cc:769] Sending connection context
0810 23:32:41.835624 (+ 336us) client_negotiation.cc:240] Negotiation successful
0810 23:32:41.836273 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13475,"thread_start_us":13189,"threads_started":1}
Aug 10 23:32:41 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404761, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
Aug 10 23:32:41 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404761, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.253@KRBTEST.COM
I20260810 23:32:41.839475 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:41.799933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:33795 (local address 127.9.48.252:34131)
0810 23:32:41.816998 (+ 17065us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:41.817003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:41.817021 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:32:41.817074 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:41.817078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:41.817149 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:41.817281 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:41.819190 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.820388 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.822514 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.823114 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.823257 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:41.827961 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:41.827971 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:41.827977 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:41.828026 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:41.832300 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.833387 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.833394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.833398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.833555 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.834180 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.834187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.834191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.834568 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:41.838704 (+ 4136us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:41.838847 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 23:32:41.839103 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16933,"thread_start_us":16805,"threads_started":1}
Aug 10 23:32:41 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.48.195@KRBTEST.COM
I20260810 23:32:41.840833 21271 heartbeater.cc:347] Connected to a master server at 127.9.48.252:34131
I20260810 23:32:41.841202 21271 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:41.842170 21271 heartbeater.cc:511] Master 127.9.48.252:34131 requested a full tablet report, sending...
I20260810 23:32:41.844090 20767 ts_manager.cc:194] Registered new tserver with Master: 2d7651a6bfd3488d88b587ed9db9ee90 (127.9.48.194:43199)
I20260810 23:32:41.847708 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:41.800460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.194:34819)
0810 23:32:41.827662 (+ 27202us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:41.827687 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:41.827863 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:41.828946 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:41.828951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:41.830432 (+ 1481us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:41.830719 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:41.830732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.832158 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.832162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:41.834125 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:41.834140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.835489 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.835498 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:41.835727 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:41.836348 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:41.836376 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:41.839664 (+ 3288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:41.844432 (+ 4768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:41.844437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:41.844441 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:41.845418 (+ 977us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:41.845925 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:41.845929 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:41.845932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:41.846094 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:41.846690 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:41.846695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:41.846854 (+ 159us) client_negotiation.cc:769] Sending connection context
0810 23:32:41.846990 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 23:32:41.847260 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26987,"mutex_wait_us":134,"thread_start_us":26575,"threads_started":1}
I20260810 23:32:41.847715 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:41.799968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:34819 (local address 127.9.48.254:40553)
0810 23:32:41.825050 (+ 25082us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:41.825059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:41.828598 (+ 3539us) server_negotiation.cc:407] Connection header received
0810 23:32:41.828668 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:41.828672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:41.828768 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:41.828891 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:41.830912 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.832009 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.834299 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.835063 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.835222 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:41.839893 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:41.839900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:41.839905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:41.839947 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:41.844276 (+ 4329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.845585 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.845605 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.845609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.845776 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.846220 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.846228 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.846233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.846570 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:41.846748 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:41.847114 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 23:32:41.847341 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24931,"thread_start_us":24783,"threads_started":1}
I20260810 23:32:41.848510 21272 heartbeater.cc:347] Connected to a master server at 127.9.48.254:40553
I20260810 23:32:41.848678 21272 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:41.849073 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:41.801284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:35727 (local address 127.9.48.194:55315)
0810 23:32:41.828408 (+ 27124us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:41.828422 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:41.829088 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:41.829502 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:41.829507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:41.830025 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:41.830266 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:41.830274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.832092 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.832097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:41.834125 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:41.834140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.835673 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.835682 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:41.835863 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:41.836499 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:41.836544 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:41.840703 (+ 4159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:41.845232 (+ 4529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:41.845235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:41.845237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:41.845750 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:41.846142 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:41.846147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:41.846151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:41.846349 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:41.847164 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:41.847170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:41.847321 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 23:32:41.847458 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 23:32:41.848581 (+ 1123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26996,"mutex_wait_us":132,"thread_start_us":26861,"threads_started":1}
I20260810 23:32:41.849071 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:41.800402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:55315 (local address 127.9.48.253:35727)
0810 23:32:41.801448 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:41.801458 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:41.828546 (+ 27088us) server_negotiation.cc:407] Connection header received
0810 23:32:41.829262 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:41.829272 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:41.829353 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:41.829472 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:41.830430 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.831907 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.834299 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.835010 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.835141 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:41.840955 (+ 5814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:41.840962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:41.840969 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:41.841014 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:41.844720 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.845885 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.845891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.845895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.846024 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.846495 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.846503 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.846506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.846921 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:41.847092 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:41.848283 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0810 23:32:41.848572 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":756,"threads_started":1}
I20260810 23:32:41.849246 21272 heartbeater.cc:511] Master 127.9.48.254:40553 requested a full tablet report, sending...
I20260810 23:32:41.850370 19465 ts_manager.cc:194] Re-registered known tserver with Master: 2d7651a6bfd3488d88b587ed9db9ee90 (127.9.48.194:43199)
I20260810 23:32:41.851004 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:41.805713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.9.48.195:41079)
0810 23:32:41.826162 (+ 20449us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:41.826174 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:41.826209 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:32:41.826286 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:41.826294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:41.826580 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:41.826768 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:41.829141 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.830813 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.832813 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:41.833851 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:41.836461 (+ 2610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:41.843049 (+ 6588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:41.843067 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:41.843099 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:41.843150 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:41.847332 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.848256 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.848266 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.848269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.848442 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:41.848919 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:41.848925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:41.848928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:41.849389 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:41.849685 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:41.850338 (+ 653us) server_negotiation.cc:299] Negotiation successful
0810 23:32:41.850610 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20250,"thread_start_us":20837,"threads_started":1}
I20260810 23:32:41.851509 19465 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:34819
I20260810 23:32:41.851895 21275 heartbeater.cc:347] Connected to a master server at 127.9.48.253:35727
I20260810 23:32:41.852032 21275 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:41.852439 20204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:41.852501 21275 heartbeater.cc:511] Master 127.9.48.253:35727 requested a full tablet report, sending...
I20260810 23:32:41.854157 20464 ts_manager.cc:194] Registered new tserver with Master: 2d7651a6bfd3488d88b587ed9db9ee90 (127.9.48.194:43199)
I20260810 23:32:41.946172 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 20004
I20260810 23:32:41.974272 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:41079
--local_ip_for_outbound_sockets=127.9.48.195
--tserver_master_addrs=127.9.48.254:40553
--webserver_port=45827
--webserver_interface=127.9.48.195
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.195.keytab
--principal=oryx/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.48.254:40553,127.9.48.253:35727
--tserver_master_addrs=127.9.48.254:40553,127.9.48.253:35727,127.9.48.252:34131 with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:32:42.200100 21052 heartbeater.cc:503] Master 127.9.48.254:40553 was elected leader, sending a full tablet report...
W20260810 23:32:42.330186 21296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:42.330459 21296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:42.330560 21296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:42.338189 21296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:42.338297 21296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:42.338418 21296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:32:42.348203 21296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.195.keytab
--principal=oryx/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:41079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=45827
--tserver_master_addrs=127.9.48.254:40553,127.9.48.253:35727,127.9.48.252:34131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:42.349594 21296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:42.351155 21296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:42.365413 21306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:42.365569 21307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:42.365700 21310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:42.366432 21296 server_base.cc:1034] running on GCE node
I20260810 23:32:42.367018 21296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:42.367974 21296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:42.369207 21296 hybrid_clock.cc:648] HybridClock initialized: now 1786404762369168 us; error 50 us; skew 500 ppm
I20260810 23:32:42.369535 21296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:42 dist-test-slave-04bb krb5kdc[19390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404762, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:42.373229 21296 init.cc:377] Logged in from keytab as oryx/127.9.48.195@KRBTEST.COM (short username oryx)
I20260810 23:32:42.374843 21296 webserver.cc:460] Webserver started at http://127.9.48.195:45827/ using document root <none> and password file <none>
I20260810 23:32:42.375574 21296 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:42.375754 21296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:42.381386 21296 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:32:42.384066 21319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:42.385437 21296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:32:42.385615 21296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "47c185c80b264d0593a81622c4ef6847"
format_stamp: "Formatted at 2026-08-10 23:32:38 on dist-test-slave-04bb"
server_key: "dcd5966dfe19b81c32e3939166e0906b"
server_key_iv: "ee2be83e649b6d3256f1d82fa66e7c2f"
server_key_version: "encryptionkey@0"
I20260810 23:32:42.386631 21296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:42.401626 21296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:42.405923 21296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:42.406225 21296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:42.407263 21296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:42.408876 21296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:42.409018 21296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:42.409149 21296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:42.409251 21296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:42.444290 21296 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:41079
I20260810 23:32:42.444378 21484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:41079 every 8 connection(s)
I20260810 23:32:42.445868 21296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:32:42.451555 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 21296
Aug 10 23:32:42 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404762, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
Aug 10 23:32:42 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404762, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.253@KRBTEST.COM
Aug 10 23:32:42 dist-test-slave-04bb krb5kdc[19390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404762, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.252@KRBTEST.COM
I20260810 23:32:42.490556 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.450767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:52983 (local address 127.9.48.254:40553)
0810 23:32:42.457036 (+ 6269us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:42.457045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:42.458023 (+ 978us) server_negotiation.cc:407] Connection header received
0810 23:32:42.458226 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:42.458235 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:42.458345 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:42.458452 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:42.461499 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.462259 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.466056 (+ 3797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.466482 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.466634 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.472425 (+ 5791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:42.472432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:42.472439 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:42.472496 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:42.484222 (+ 11726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:42.487022 (+ 2800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:42.487030 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:42.487035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:42.487216 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:42.488110 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:42.488116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:42.488118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:42.488481 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:42.488656 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:42.489877 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0810 23:32:42.490144 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6118,"thread_start_us":6014,"threads_started":1}
I20260810 23:32:42.491351 21501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.450196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.195:52983)
0810 23:32:42.457902 (+ 7706us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:42.457918 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:42.458076 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:42.459638 (+ 1562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:42.459642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:42.460240 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:42.461309 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.461336 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.462793 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.462814 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.465938 (+ 3124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.465951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.466729 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.466736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.467095 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.467991 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:42.468032 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:42.471978 (+ 3946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:42.484366 (+ 12388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:42.484381 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:42.484403 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:42.485154 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:42.487434 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:42.487441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:42.487447 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:42.487689 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:42.488642 (+ 953us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:42.488651 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:42.489007 (+ 356us) client_negotiation.cc:769] Sending connection context
0810 23:32:42.489302 (+ 295us) client_negotiation.cc:240] Negotiation successful
0810 23:32:42.489858 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7598,"mutex_wait_us":268,"thread_start_us":1898,"threads_started":1}
I20260810 23:32:42.491770 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.448997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:57357 (local address 127.9.48.253:35727)
0810 23:32:42.459954 (+ 10957us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:42.459963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:42.459990 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:32:42.460070 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:42.460078 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:42.460178 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:42.460319 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:42.462465 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.463623 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.466037 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.466649 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.466790 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.479163 (+ 12373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:42.479171 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:42.479176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:42.479221 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:42.486171 (+ 6950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:42.487379 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:42.487390 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:42.487395 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:42.487561 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:42.488665 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:42.488674 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:42.488680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:42.489163 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:42.490217 (+ 1054us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:42.490409 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 23:32:42.491168 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10773,"thread_start_us":10648,"threads_started":1}
I20260810 23:32:42.492273 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.449608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:35727 (local address 127.9.48.195:57357)
0810 23:32:42.459352 (+ 9744us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:42.459372 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:42.459478 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:42.460580 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:42.460588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:42.461798 (+ 1210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:42.462204 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.462220 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.463800 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.463806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.465842 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.465857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.466886 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.466892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.467101 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.468277 (+ 1176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:42.468306 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:42.478887 (+ 10581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:42.486339 (+ 7452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:42.486345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:42.486350 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:42.486880 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:42.487749 (+ 869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:42.487754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:42.487758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:42.487961 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:42.489305 (+ 1344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:42.489309 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:42.489447 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 23:32:42.489584 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 23:32:42.491855 (+ 2271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9604,"mutex_wait_us":3534,"thread_start_us":6729,"threads_started":1}
I20260810 23:32:42.494266 21486 heartbeater.cc:347] Connected to a master server at 127.9.48.254:40553
I20260810 23:32:42.494618 21486 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:42.495608 21486 heartbeater.cc:511] Master 127.9.48.254:40553 requested a full tablet report, sending...
I20260810 23:32:42.497378 19465 ts_manager.cc:194] Re-registered known tserver with Master: 47c185c80b264d0593a81622c4ef6847 (127.9.48.195:41079)
I20260810 23:32:42.498526 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.467906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32976 (local address 127.9.48.254:40553)
0810 23:32:42.472149 (+ 4243us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:42.472164 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:42.472181 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:32:42.472231 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:42.472236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:42.472318 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:42.475511 (+ 3193us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:42.475735 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.477047 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.479474 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.480237 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.480379 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.485480 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:42.485490 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:42.485496 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:42.485533 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:42.494598 (+ 9065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:42.495593 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:42.495598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:42.495601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:42.495747 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:42.496245 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:42.496250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:42.496253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:42.496602 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:42.496858 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:42.497869 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0810 23:32:42.498094 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2880,"thread_start_us":2727,"threads_started":1}
I20260810 23:32:42.498561 19465 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:52983
I20260810 23:32:42.495626 21487 heartbeater.cc:347] Connected to a master server at 127.9.48.253:35727
I20260810 23:32:42.499042 21487 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:42.499678 21487 heartbeater.cc:511] Master 127.9.48.253:35727 requested a full tablet report, sending...
I20260810 23:32:42.501397 20464 ts_manager.cc:194] Registered new tserver with Master: 47c185c80b264d0593a81622c4ef6847 (127.9.48.195:41079)
I20260810 23:32:42.501870 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.449608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:34131 (local address 127.9.48.195:43451)
0810 23:32:42.452581 (+ 2973us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:42.452639 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:42.455796 (+ 3157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:42.459138 (+ 3342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:42.459165 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:42.460131 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:42.462460 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.462474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.463877 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.463881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.466996 (+ 3115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.467016 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.468284 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.468293 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.468601 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.472899 (+ 4298us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:42.472932 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:42.476471 (+ 3539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:42.497897 (+ 21426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:42.497902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:42.497905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:42.498423 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:42.499305 (+ 882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:42.499313 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:42.499322 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:42.499550 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:42.500260 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:42.500267 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:42.500447 (+ 180us) client_negotiation.cc:769] Sending connection context
0810 23:32:42.500617 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 23:32:42.501241 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2648,"mutex_wait_us":1280,"spinlock_wait_cycles":16768,"thread_start_us":9320,"threads_started":1}
I20260810 23:32:42.502887 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.450702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:43451 (local address 127.9.48.252:34131)
0810 23:32:42.458527 (+ 7825us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:42.458533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:42.458551 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:32:42.458620 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:42.458625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:42.458701 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:42.458823 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:42.462642 (+ 3819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.463712 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.467189 (+ 3477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.467847 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.467975 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.476961 (+ 8986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:42.476971 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:42.476979 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:42.477038 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:42.497721 (+ 20683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:42.498886 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:42.498894 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:42.498897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:42.499165 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:42.499716 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:42.499722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:42.499725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:42.500086 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:42.502120 (+ 2034us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:42.502289 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 23:32:42.502548 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7706,"thread_start_us":7595,"threads_started":1}
I20260810 23:32:42.504644 21485 heartbeater.cc:347] Connected to a master server at 127.9.48.252:34131
I20260810 23:32:42.504784 21485 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:42.505209 21485 heartbeater.cc:511] Master 127.9.48.252:34131 requested a full tablet report, sending...
I20260810 23:32:42.506376 20767 ts_manager.cc:194] Registered new tserver with Master: 47c185c80b264d0593a81622c4ef6847 (127.9.48.195:41079)
I20260810 23:32:42.503690 19466 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:42.513598 19466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:42.516600 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.470270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.9.48.252:34131)
0810 23:32:42.495239 (+ 24969us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:42.495246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:42.495263 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:32:42.495805 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:42.495810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:42.495912 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:42.496017 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:42.500187 (+ 4170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.501591 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.504706 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.505358 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.505490 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.509497 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:42.509502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:42.509505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:42.509567 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:42.513276 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:42.514245 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:42.514252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:42.514257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:42.514418 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:42.514908 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:42.514916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:42.514921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:42.515261 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:42.515517 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:42.515942 (+ 425us) server_negotiation.cc:299] Negotiation successful
0810 23:32:42.516166 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24832,"thread_start_us":24942,"threads_started":1}
I20260810 23:32:42.523334 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.472176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.9.48.253:35727)
0810 23:32:42.481512 (+ 9336us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:42.481519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:42.495350 (+ 13831us) server_negotiation.cc:407] Connection header received
0810 23:32:42.495407 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:42.495411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:42.495479 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:42.495578 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:42.507130 (+ 11552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.508638 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.510713 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.511524 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.511674 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.514915 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:42.514920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:42.514923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:42.514953 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:42.518904 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:42.519826 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:42.519832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:42.519835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:42.520038 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:42.520514 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:42.520520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:42.520556 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:42.520966 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:42.522286 (+ 1320us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:42.522466 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 23:32:42.522705 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9191,"thread_start_us":9190,"threads_started":1}
I20260810 23:32:42.557478 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.540842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47792 (local address 127.9.48.193:46437)
0810 23:32:42.547528 (+ 6686us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:42.547541 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:42.550003 (+ 2462us) server_negotiation.cc:407] Connection header received
0810 23:32:42.550173 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:42.550185 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:42.550501 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:42.550724 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:42.551168 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.552763 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.554856 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.556450 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.556637 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.556846 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:42.557117 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 23:32:42.557265 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6501,"thread_start_us":6388,"threads_started":1}
I20260810 23:32:42.557509 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.540107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46437 (local address 127.0.0.1:47792)
0810 23:32:42.549871 (+ 9764us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:42.549894 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:42.550021 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:42.550711 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:42.550714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:42.550736 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:42.550982 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.550990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.552897 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.552903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.554714 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.554725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.556713 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.556720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.556896 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.556934 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 23:32:42.557030 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 23:32:42.557159 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9600,"thread_start_us":7940,"threads_started":1}
I20260810 23:32:42.561414 21532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.537755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:41079 (local address 127.0.0.1:57470)
0810 23:32:42.544890 (+ 7135us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:42.544915 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:42.545059 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:42.551305 (+ 6246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:42.551310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:42.551339 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:42.551708 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.551721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.554876 (+ 3155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.554883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.556986 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.557020 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.560513 (+ 3493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.560517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.560764 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.560800 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:32:42.560900 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 23:32:42.561046 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6620,"thread_start_us":645,"threads_started":1}
I20260810 23:32:42.561465 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.538118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.9.48.195:41079)
0810 23:32:42.550630 (+ 12512us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:42.550641 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:42.550676 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:32:42.550772 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:42.550782 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:42.551077 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:42.551333 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:42.551939 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.554711 (+ 2772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.558773 (+ 4062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.560317 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.560453 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.560680 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:42.560960 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 23:32:42.561120 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12326,"thread_start_us":12142,"threads_started":1}
I20260810 23:32:42.561818 20942 tablet_service.cc:1467] Processing CreateTablet for tablet 01b7b960a7ec4797928f98f44a928517 (DEFAULT_TABLE table=test-table [id=ca9b2872569747eaa05d39dfdf83a3a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:42.563385 20942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b7b960a7ec4797928f98f44a928517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:42.563692 21538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.538223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:43199 (local address 127.0.0.1:44730)
0810 23:32:42.548334 (+ 10111us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:42.548353 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:42.548469 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:42.553910 (+ 5441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:42.553915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:42.553940 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:42.554305 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.554317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.555709 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.555717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.560936 (+ 5219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.560949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.562795 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.562801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.563002 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.563034 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:32:42.563133 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 23:32:42.563256 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9962,"thread_start_us":11279,"threads_started":1}
I20260810 23:32:42.563836 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.538452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44730 (local address 127.9.48.194:43199)
0810 23:32:42.553302 (+ 14850us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:42.553312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:42.553343 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:32:42.553447 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:42.553455 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:42.553695 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:42.553903 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:42.554478 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.555571 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.561141 (+ 5570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.562571 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.562712 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.562924 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:42.563361 (+ 437us) server_negotiation.cc:299] Negotiation successful
0810 23:32:42.563530 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14670,"thread_start_us":15580,"threads_started":1}
I20260810 23:32:42.565479 21390 tablet_service.cc:1467] Processing CreateTablet for tablet 01b7b960a7ec4797928f98f44a928517 (DEFAULT_TABLE table=test-table [id=ca9b2872569747eaa05d39dfdf83a3a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:42.566607 21390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b7b960a7ec4797928f98f44a928517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:42.567672 21168 tablet_service.cc:1467] Processing CreateTablet for tablet 01b7b960a7ec4797928f98f44a928517 (DEFAULT_TABLE table=test-table [id=ca9b2872569747eaa05d39dfdf83a3a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:42.569106 21168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b7b960a7ec4797928f98f44a928517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:42.579360 21540 tablet_bootstrap.cc:492] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847: Bootstrap starting.
I20260810 23:32:42.581272 21541 tablet_bootstrap.cc:492] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224: Bootstrap starting.
I20260810 23:32:42.583575 21540 tablet_bootstrap.cc:654] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:42.584746 21541 tablet_bootstrap.cc:654] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:42.585355 21540 log.cc:824] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:42.585449 21542 tablet_bootstrap.cc:492] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90: Bootstrap starting.
I20260810 23:32:42.586067 21541 log.cc:824] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:42.588517 21542 tablet_bootstrap.cc:654] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:42.589692 21541 tablet_bootstrap.cc:492] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224: No bootstrap required, opened a new log
I20260810 23:32:42.590065 21541 ts_tablet_manager.cc:1397] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260810 23:32:42.591215 21542 log.cc:824] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:42.591276 21540 tablet_bootstrap.cc:492] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847: No bootstrap required, opened a new log
I20260810 23:32:42.591536 21540 ts_tablet_manager.cc:1397] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260810 23:32:42.593683 21542 tablet_bootstrap.cc:492] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90: No bootstrap required, opened a new log
I20260810 23:32:42.593961 21542 ts_tablet_manager.cc:1397] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20260810 23:32:42.601118 21541 raft_consensus.cc:348] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } }
I20260810 23:32:42.601562 21541 raft_consensus.cc:374] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:42.601702 21541 raft_consensus.cc:729] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94048588350439e91217f75d0058224, State: Initialized, Role: FOLLOWER
I20260810 23:32:42.601521 21542 raft_consensus.cc:348] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } }
I20260810 23:32:42.602092 21542 raft_consensus.cc:374] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:42.602213 21542 raft_consensus.cc:729] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7651a6bfd3488d88b587ed9db9ee90, State: Initialized, Role: FOLLOWER
I20260810 23:32:42.602205 21540 raft_consensus.cc:348] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } }
I20260810 23:32:42.602433 21541 consensus_queue.cc:260] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } }
I20260810 23:32:42.602646 21540 raft_consensus.cc:374] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:42.602743 21540 raft_consensus.cc:729] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47c185c80b264d0593a81622c4ef6847, State: Initialized, Role: FOLLOWER
I20260810 23:32:42.602895 21542 consensus_queue.cc:260] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } }
I20260810 23:32:42.603331 21540 consensus_queue.cc:260] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } }
I20260810 23:32:42.604353 21541 ts_tablet_manager.cc:1428] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260810 23:32:42.605072 21540 ts_tablet_manager.cc:1428] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260810 23:32:42.605522 21486 heartbeater.cc:503] Master 127.9.48.254:40553 was elected leader, sending a full tablet report...
I20260810 23:32:42.610530 21542 ts_tablet_manager.cc:1428] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
I20260810 23:32:42.610877 21272 heartbeater.cc:503] Master 127.9.48.254:40553 was elected leader, sending a full tablet report...
W20260810 23:32:42.648202 21056 tablet.cc:2369] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:32:42.707834 21490 tablet.cc:2369] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:32:42.825353 21276 tablet.cc:2369] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:32:42.902676 21546 raft_consensus.cc:482] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:42.903000 21546 raft_consensus.cc:504] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } }
I20260810 23:32:42.904786 21546 leader_election.cc:288] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47c185c80b264d0593a81622c4ef6847 (127.9.48.195:41079), 2d7651a6bfd3488d88b587ed9db9ee90 (127.9.48.194:43199)
I20260810 23:32:42.913931 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.905348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:38069 (local address 127.9.48.194:43199)
0810 23:32:42.905538 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:42.905544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:42.906289 (+ 745us) server_negotiation.cc:407] Connection header received
0810 23:32:42.906529 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:42.906536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:42.906613 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:42.906725 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:42.907300 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.908346 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.911472 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.912757 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.912936 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.913007 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:42.913534 (+ 527us) server_negotiation.cc:299] Negotiation successful
0810 23:32:42.913665 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:32:42.913932 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.905317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:56645 (local address 127.9.48.195:41079)
0810 23:32:42.905520 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:42.905527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:42.905979 (+ 452us) server_negotiation.cc:407] Connection header received
0810 23:32:42.906252 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:42.906256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:42.906321 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:42.906423 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:42.907077 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.908216 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.911473 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.912755 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.912931 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.913007 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:42.913534 (+ 527us) server_negotiation.cc:299] Negotiation successful
0810 23:32:42.913665 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:32:42.914310 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.905069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:41079 (local address 127.9.48.193:56645)
0810 23:32:42.905855 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:42.905874 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:42.906079 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:42.906442 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:42.906447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:42.906475 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:42.906915 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.906931 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.908353 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.908357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.911301 (+ 2944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.911313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.913060 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.913068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.913294 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.913388 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 23:32:42.913491 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:32:42.913706 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":311,"thread_start_us":450,"threads_started":1}
I20260810 23:32:42.914409 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:42.905069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:43199 (local address 127.9.48.193:38069)
0810 23:32:42.906177 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:42.906192 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:42.906355 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:42.906748 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:42.906753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:42.906778 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:42.907139 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.907149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.908497 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.908507 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.911297 (+ 2790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:42.911313 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:42.913077 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:42.913082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:42.913278 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:42.913387 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 23:32:42.913491 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 23:32:42.913702 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"mutex_wait_us":656,"thread_start_us":348,"threads_started":1}
I20260810 23:32:42.915359 21417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b7b960a7ec4797928f98f44a928517" candidate_uuid: "c94048588350439e91217f75d0058224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c185c80b264d0593a81622c4ef6847" is_pre_election: true
I20260810 23:32:42.915606 21198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b7b960a7ec4797928f98f44a928517" candidate_uuid: "c94048588350439e91217f75d0058224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" is_pre_election: true
I20260810 23:32:42.915947 21417 raft_consensus.cc:2454] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c94048588350439e91217f75d0058224 in term 0.
I20260810 23:32:42.916142 21198 raft_consensus.cc:2454] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c94048588350439e91217f75d0058224 in term 0.
I20260810 23:32:42.916846 20877 leader_election.cc:302] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d7651a6bfd3488d88b587ed9db9ee90, 47c185c80b264d0593a81622c4ef6847, c94048588350439e91217f75d0058224; no voters:
I20260810 23:32:42.917207 21546 raft_consensus.cc:2781] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:42.917347 21546 raft_consensus.cc:482] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:42.917477 21546 raft_consensus.cc:3037] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:42.919703 21546 raft_consensus.cc:504] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } }
I20260810 23:32:42.920320 21546 leader_election.cc:288] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [CANDIDATE]: Term 1 election: Requested vote from peers 47c185c80b264d0593a81622c4ef6847 (127.9.48.195:41079), 2d7651a6bfd3488d88b587ed9db9ee90 (127.9.48.194:43199)
I20260810 23:32:42.920888 21198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b7b960a7ec4797928f98f44a928517" candidate_uuid: "c94048588350439e91217f75d0058224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7651a6bfd3488d88b587ed9db9ee90"
I20260810 23:32:42.920892 21417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b7b960a7ec4797928f98f44a928517" candidate_uuid: "c94048588350439e91217f75d0058224" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c185c80b264d0593a81622c4ef6847"
I20260810 23:32:42.921103 21417 raft_consensus.cc:3037] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:42.921104 21198 raft_consensus.cc:3037] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:42.924396 21417 raft_consensus.cc:2454] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c94048588350439e91217f75d0058224 in term 1.
I20260810 23:32:42.924396 21198 raft_consensus.cc:2454] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c94048588350439e91217f75d0058224 in term 1.
I20260810 23:32:42.924885 20874 leader_election.cc:302] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d7651a6bfd3488d88b587ed9db9ee90, 47c185c80b264d0593a81622c4ef6847, c94048588350439e91217f75d0058224; no voters:
I20260810 23:32:42.925315 21546 raft_consensus.cc:2781] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:42.925776 21546 raft_consensus.cc:686] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [term 1 LEADER]: Becoming Leader. State: Replica: c94048588350439e91217f75d0058224, State: Running, Role: LEADER
I20260810 23:32:42.926342 21546 consensus_queue.cc:237] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } }
I20260810 23:32:42.931491 19466 catalog_manager.cc:5495] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 reported cstate change: term changed from 0 to 1, leader changed from <none> to c94048588350439e91217f75d0058224 (127.9.48.193). New cstate: current_term: 1 leader_uuid: "c94048588350439e91217f75d0058224" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } health_report { overall_health: HEALTHY } } }
I20260810 23:32:43.017194 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:43.011666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.9.48.193:46437)
0810 23:32:43.011886 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:43.011892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:43.011912 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:32:43.012116 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:43.012121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:43.012224 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:43.012390 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:32:43.012800 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:43.013518 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:43.014873 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:43.015321 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:43.015463 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:43.015947 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:32:43.016532 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:32:43.016710 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:43.016804 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 23:32:43.016872 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:32:43.025293 21417 raft_consensus.cc:1261] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847 [term 1 FOLLOWER]: Refusing update from remote peer c94048588350439e91217f75d0058224: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:43.025360 21198 raft_consensus.cc:1261] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 1 FOLLOWER]: Refusing update from remote peer c94048588350439e91217f75d0058224: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:43.026353 21554 consensus_queue.cc:1035] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:43.026765 21546 consensus_queue.cc:1035] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:43.053893 21563 mvcc.cc:205] Tried to move back new op lower bound from 7317113909341442048 to 7317113908950122496. Current Snapshot: MvccSnapshot[applied={T|T < 7317113909341442048}]
I20260810 23:32:43.056318 19466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:43.056633 19466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:43.064289 21390 tablet_service.cc:1514] Processing DeleteTablet for tablet 01b7b960a7ec4797928f98f44a928517 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:57470
I20260810 23:32:43.065042 19466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2724aeacf453402d98aecdbb20891190: Sending DeleteTablet for 3 replicas of tablet 01b7b960a7ec4797928f98f44a928517
I20260810 23:32:43.065171 21577 tablet_replica.cc:330] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847: stopping tablet replica
I20260810 23:32:43.065769 21390 tablet_service.cc:1514] Processing DeleteTablet for tablet 01b7b960a7ec4797928f98f44a928517 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:57470
I20260810 23:32:43.065860 21577 raft_consensus.cc:2229] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:43.066138 21168 tablet_service.cc:1514] Processing DeleteTablet for tablet 01b7b960a7ec4797928f98f44a928517 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:44730
I20260810 23:32:43.066293 21577 raft_consensus.cc:2258] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:43.066874 20942 tablet_service.cc:1514] Processing DeleteTablet for tablet 01b7b960a7ec4797928f98f44a928517 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:43 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:47792
I20260810 23:32:43.066992 21578 tablet_replica.cc:330] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90: stopping tablet replica
I20260810 23:32:43.067370 21578 raft_consensus.cc:2229] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:43.067915 21578 raft_consensus.cc:2258] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:43.068476 21579 tablet_replica.cc:330] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224: stopping tablet replica
I20260810 23:32:43.068904 21579 raft_consensus.cc:2229] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:43.069260 21577 ts_tablet_manager.cc:1905] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:43.069553 21579 raft_consensus.cc:2258] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:43.078153 21577 ts_tablet_manager.cc:1918] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:43.078387 21577 log.cc:1196] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/01b7b960a7ec4797928f98f44a928517
I20260810 23:32:43.079020 21577 ts_tablet_manager.cc:1939] T 01b7b960a7ec4797928f98f44a928517 P 47c185c80b264d0593a81622c4ef6847: Deleting consensus metadata
I20260810 23:32:43.080206 21579 ts_tablet_manager.cc:1905] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:32:43.081708 19437 catalog_manager.cc:4808] TS 47c185c80b264d0593a81622c4ef6847 (127.9.48.195:41079): delete failed for tablet 01b7b960a7ec4797928f98f44a928517 because the tablet was not found. No further retry: Not found: Tablet not found: 01b7b960a7ec4797928f98f44a928517
I20260810 23:32:43.082417 21578 ts_tablet_manager.cc:1905] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:43.088904 21578 ts_tablet_manager.cc:1918] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:43.089088 21578 log.cc:1196] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/01b7b960a7ec4797928f98f44a928517
I20260810 23:32:43.089776 21578 ts_tablet_manager.cc:1939] T 01b7b960a7ec4797928f98f44a928517 P 2d7651a6bfd3488d88b587ed9db9ee90: Deleting consensus metadata
I20260810 23:32:43.092235 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 19405
I20260810 23:32:43.093367 19511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:43.097487 21579 ts_tablet_manager.cc:1918] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:43.098768 21579 log.cc:1196] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/01b7b960a7ec4797928f98f44a928517
I20260810 23:32:43.099556 21579 ts_tablet_manager.cc:1939] T 01b7b960a7ec4797928f98f44a928517 P c94048588350439e91217f75d0058224: Deleting consensus metadata
I20260810 23:32:43.101951 19433 catalog_manager.cc:4841] TS 2d7651a6bfd3488d88b587ed9db9ee90 (127.9.48.194:43199): tablet 01b7b960a7ec4797928f98f44a928517 (table test-table [id=ca9b2872569747eaa05d39dfdf83a3a4]) successfully deleted
I20260810 23:32:43.116920 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:43.070472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37378 (local address 127.9.48.253:35727)
0810 23:32:43.098107 (+ 27635us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:43.098116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:43.098142 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:32:43.098218 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:43.098227 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:43.098352 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:43.098488 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:43.100621 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:43.102020 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:43.104207 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:43.104911 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:43.105058 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:43.111227 (+ 6169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:43.111233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:43.111236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:43.111294 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:43.114164 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:43.114842 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:43.114849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:43.114854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:43.114979 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:43.115364 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:43.115372 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:43.115375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:43.115648 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:43.115946 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:43.116276 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 23:32:43.116508 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27463,"thread_start_us":27293,"threads_started":1}
I20260810 23:32:43.130131 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:43.091337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.9.48.252:34131)
0810 23:32:43.102679 (+ 11342us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:43.102687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:43.117017 (+ 14330us) server_negotiation.cc:407] Connection header received
0810 23:32:43.117276 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:43.117282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:43.117392 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:43.117480 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:43.119378 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:43.120113 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:43.121439 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:43.121833 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:43.122002 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:43.124480 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:43.124485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:43.124488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:43.124565 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:43.127420 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:43.128065 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:43.128072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:43.128078 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:43.128215 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:43.128645 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:43.128651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:43.128654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:43.128947 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:43.129206 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:43.129537 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 23:32:43.129768 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11157,"thread_start_us":11043,"threads_started":1}
I20260810 23:32:43.269131 19435 catalog_manager.cc:4841] TS c94048588350439e91217f75d0058224 (127.9.48.193:46437): tablet 01b7b960a7ec4797928f98f44a928517 (table test-table [id=ca9b2872569747eaa05d39dfdf83a3a4]) successfully deleted
W20260810 23:32:43.270390 19435 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.9.48.193:46437, user_credentials={real_user=slave}} blocked reactor thread for 167834us
I20260810 23:32:43.271530 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 19405
I20260810 23:32:43.322338 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:43.304219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.9.48.252:34131)
0810 23:32:43.304415 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:43.304421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:43.305315 (+ 894us) server_negotiation.cc:407] Connection header received
0810 23:32:43.305581 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:43.305587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:43.305723 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:43.305807 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:43.308274 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:43.309508 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:43.311611 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:43.312249 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:43.312379 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:43.316051 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:43.316059 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:43.316065 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:43.316110 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:43.319183 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:43.319859 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:43.319866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:43.319869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:43.319995 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:43.320503 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:43.320508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:43.320511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:43.320956 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:43.321131 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:43.321742 (+ 611us) server_negotiation.cc:299] Negotiation successful
0810 23:32:43.321989 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:32:43.323884 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:43.304815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.9.48.253:35727)
0810 23:32:43.305008 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:43.305013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:43.307398 (+ 2385us) server_negotiation.cc:407] Connection header received
0810 23:32:43.307716 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:43.307721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:43.307785 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:43.307856 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:43.310485 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:43.311300 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:43.313099 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:43.313601 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:43.313763 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:43.316446 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:43.316454 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:43.316458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:43.316501 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:43.320139 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:43.320955 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:43.320962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:43.320966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:43.321122 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:43.321584 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:43.321590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:43.321593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:43.321952 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:43.322189 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:43.323090 (+ 901us) server_negotiation.cc:299] Negotiation successful
0810 23:32:43.323341 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:32:44.048981 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.048183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.194:35143)
0810 23:32:44.048688 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.048813 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":238,"threads_started":1}
W20260810 23:32:44.049568 21272 heartbeater.cc:644] Failed to heartbeat to 127.9.48.254:40553 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
I20260810 23:32:44.053623 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.052803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.193:39237)
0810 23:32:44.053314 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.053469 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":233,"threads_started":1}
W20260810 23:32:44.053989 21052 heartbeater.cc:644] Failed to heartbeat to 127.9.48.254:40553 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
I20260810 23:32:44.066190 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.065429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.195:50473)
0810 23:32:44.065899 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.066016 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":254,"threads_started":1}
W20260810 23:32:44.066594 21486 heartbeater.cc:644] Failed to heartbeat to 127.9.48.254:40553 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
I20260810 23:32:44.563957 21596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2724aeacf453402d98aecdbb20891190)
I20260810 23:32:44.564280 21596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } }
I20260810 23:32:44.564393 21598 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2724aeacf453402d98aecdbb20891190)
I20260810 23:32:44.564683 21598 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } }
I20260810 23:32:44.565832 21596 leader_election.cc:288] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2724aeacf453402d98aecdbb20891190 (127.9.48.254:40553), ed7ac368f3fe4686b6349f2599e4d93f (127.9.48.253:35727)
I20260810 23:32:44.566042 21598 leader_election.cc:288] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2724aeacf453402d98aecdbb20891190 (127.9.48.254:40553), 1a4caf350fc747079975acc1aceb6a3c (127.9.48.252:34131)
I20260810 23:32:44.567325 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.565916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:33350)
0810 23:32:44.566744 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.566976 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":664,"threads_started":1}
I20260810 23:32:44.571483 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.568799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:33356)
0810 23:32:44.570735 (+ 1936us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.571012 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1755,"thread_start_us":1569,"threads_started":1}
W20260810 23:32:44.571916 20437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
W20260810 23:32:44.572803 20733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
I20260810 23:32:44.574000 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.573453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:33358)
0810 23:32:44.573661 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.573777 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 23:32:44.574467 20437 leader_election.cc:334] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2724aeacf453402d98aecdbb20891190 (127.9.48.254:40553): Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
I20260810 23:32:44.575640 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.566104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:35727 (local address 127.0.0.1:37396)
0810 23:32:44.567695 (+ 1591us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.567721 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:44.567913 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:44.568932 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:44.568936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:44.568959 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:44.569174 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:44.569182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.570845 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.570852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:44.573084 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:44.573099 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.574886 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.574892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:44.575088 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.575144 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 23:32:44.575249 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 23:32:44.575380 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"thread_start_us":555,"threads_started":1}
I20260810 23:32:44.576196 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.566515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.9.48.253:35727)
0810 23:32:44.568566 (+ 2051us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:44.568573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:44.568593 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:32:44.568668 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:44.568672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:44.568766 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:44.569286 (+ 520us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:44.569474 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.570655 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.573257 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.574545 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.574656 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.574750 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:44.575734 (+ 984us) server_negotiation.cc:299] Negotiation successful
0810 23:32:44.575874 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1868,"thread_start_us":1668,"threads_started":1}
I20260810 23:32:44.576448 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.575853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:33368)
0810 23:32:44.576020 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.576202 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:32:44.576615 20733 leader_election.cc:334] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2724aeacf453402d98aecdbb20891190 (127.9.48.254:40553): Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
I20260810 23:32:44.576845 20486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a4caf350fc747079975acc1aceb6a3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" is_pre_election: true
I20260810 23:32:44.577219 20486 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a4caf350fc747079975acc1aceb6a3c in term 1.
I20260810 23:32:44.577792 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.566462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:34131 (local address 127.0.0.1:58622)
0810 23:32:44.568675 (+ 2213us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.568690 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:44.568785 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:44.570593 (+ 1808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:44.570601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:44.570634 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:44.571197 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:44.571213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.572939 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.572946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:44.575142 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:44.575157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.576987 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.576992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:44.577194 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.577222 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 23:32:44.577300 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 23:32:44.577440 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2101,"thread_start_us":1948,"threads_started":1}
I20260810 23:32:44.577934 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.567046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.9.48.252:34131)
0810 23:32:44.570283 (+ 3237us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:44.570288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:44.570325 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:32:44.570379 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:44.570383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:44.570443 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:44.571341 (+ 898us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:44.571616 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.572739 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.575287 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.576738 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.576924 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.577026 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:44.577374 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 23:32:44.577509 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3131,"thread_start_us":2994,"threads_started":1}
I20260810 23:32:44.577824 20738 leader_election.cc:302] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a4caf350fc747079975acc1aceb6a3c, ed7ac368f3fe4686b6349f2599e4d93f; no voters: 2724aeacf453402d98aecdbb20891190
I20260810 23:32:44.578359 21596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 23:32:44.578495 21596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2724aeacf453402d98aecdbb20891190)
I20260810 23:32:44.578624 21596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:32:44.578630 20783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1a4caf350fc747079975acc1aceb6a3c" is_pre_election: true
I20260810 23:32:44.580946 21596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } }
I20260810 23:32:44.581489 20783 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ed7ac368f3fe4686b6349f2599e4d93f in current term 2: Already voted for candidate 1a4caf350fc747079975acc1aceb6a3c in this term.
I20260810 23:32:44.581789 21596 leader_election.cc:288] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [CANDIDATE]: Term 2 election: Requested vote from peers 2724aeacf453402d98aecdbb20891190 (127.9.48.254:40553), ed7ac368f3fe4686b6349f2599e4d93f (127.9.48.253:35727)
I20260810 23:32:44.582103 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.581683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:33372)
0810 23:32:44.581912 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.581999 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260810 23:32:44.582122 20486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a4caf350fc747079975acc1aceb6a3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ed7ac368f3fe4686b6349f2599e4d93f"
I20260810 23:32:44.582348 20486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:32:44.582715 20436 leader_election.cc:302] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed7ac368f3fe4686b6349f2599e4d93f; no voters: 1a4caf350fc747079975acc1aceb6a3c, 2724aeacf453402d98aecdbb20891190
I20260810 23:32:44.583120 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.582745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:33384)
0810 23:32:44.582888 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.582960 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:32:44.583293 20733 leader_election.cc:334] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2724aeacf453402d98aecdbb20891190 (127.9.48.254:40553): Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
I20260810 23:32:44.584893 20486 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a4caf350fc747079975acc1aceb6a3c in term 2.
I20260810 23:32:44.585124 21598 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:32:44.585333 20738 leader_election.cc:302] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a4caf350fc747079975acc1aceb6a3c, ed7ac368f3fe4686b6349f2599e4d93f; no voters: 2724aeacf453402d98aecdbb20891190
I20260810 23:32:44.585661 21596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:32:44.586112 21596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [term 2 LEADER]: Becoming Leader. State: Replica: 1a4caf350fc747079975acc1aceb6a3c, State: Running, Role: LEADER
I20260810 23:32:44.586493 21596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } }
I20260810 23:32:44.588124 21612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a4caf350fc747079975acc1aceb6a3c. Latest consensus state: current_term: 2 leader_uuid: "1a4caf350fc747079975acc1aceb6a3c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } } }
I20260810 23:32:44.588627 21612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:44.589300 21614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:44.593441 21614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:44.594247 21614 catalog_manager.cc:1259] Loaded cluster ID: e341c7ce902047f2843c6a47b0eb866a
I20260810 23:32:44.594319 21614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:44.595489 21614 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:44.596345 21614 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: Loaded TSK: 0
I20260810 23:32:44.597268 21614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:32:44.603322 20767 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:44.610754 20767 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:44.614787 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.614366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:33388)
0810 23:32:44.614550 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.614635 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:32:44.614893 20486 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [term 2 FOLLOWER]: Refusing update from remote peer 1a4caf350fc747079975acc1aceb6a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 23:32:44.615756 21612 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 23:32:44.615909 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.615500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:33404)
0810 23:32:44.615685 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.615757 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 23:32:44.616189 20733 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c -> Peer 2724aeacf453402d98aecdbb20891190 (127.9.48.254:40553): Couldn't send request to peer 2724aeacf453402d98aecdbb20891190. Status: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:32:44.618351 21598 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a4caf350fc747079975acc1aceb6a3c. Latest consensus state: current_term: 2 leader_uuid: "1a4caf350fc747079975acc1aceb6a3c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } } }
I20260810 23:32:44.618592 21598 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:44.620249 21612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "1a4caf350fc747079975acc1aceb6a3c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } } }
I20260810 23:32:44.620605 21612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:44.620977 21598 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "1a4caf350fc747079975acc1aceb6a3c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2724aeacf453402d98aecdbb20891190" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40553 } } peers { permanent_uuid: "ed7ac368f3fe4686b6349f2599e4d93f" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 35727 } attrs { promote: false } } peers { permanent_uuid: "1a4caf350fc747079975acc1aceb6a3c" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 34131 } attrs { promote: false } } }
I20260810 23:32:44.621309 21598 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7ac368f3fe4686b6349f2599e4d93f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:32:44.638298 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.629931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:43199 (local address 127.0.0.1:44742)
0810 23:32:44.630165 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.630183 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:44.630279 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:44.630784 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:44.630811 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:44.630837 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:44.631074 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:44.631093 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.633093 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.633103 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:44.635806 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:44.635829 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.637513 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.637522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:44.637801 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.637835 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:32:44.637946 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 23:32:44.638058 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260810 23:32:44.638928 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.630050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.9.48.194:43199)
0810 23:32:44.630517 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:44.630522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:44.630538 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:32:44.630599 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:44.630602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:44.630658 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:44.630739 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:44.631315 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.632874 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.636173 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.637269 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.637418 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.637545 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:44.638534 (+ 989us) server_negotiation.cc:299] Negotiation successful
0810 23:32:44.638684 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":269,"threads_started":1}
I20260810 23:32:44.639714 21168 tablet_service.cc:1467] Processing CreateTablet for tablet e790a08c8f484746b6b8fd51e682ac35 (DEFAULT_TABLE table=test-table [id=47aa5a19dde9410ea0cce9e687675355]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:44.640208 21168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e790a08c8f484746b6b8fd51e682ac35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:44.642056 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.631632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.9.48.195:41079)
0810 23:32:44.632939 (+ 1307us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:44.632949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:44.633721 (+ 772us) server_negotiation.cc:407] Connection header received
0810 23:32:44.633789 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:44.633794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:44.633855 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:44.633961 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:44.634626 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.635736 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.638521 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.639672 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.639790 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.639910 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:44.641641 (+ 1731us) server_negotiation.cc:299] Negotiation successful
0810 23:32:44.641790 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1153,"thread_start_us":1054,"threads_started":1}
I20260810 23:32:44.642831 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.631196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:41079 (local address 127.0.0.1:57484)
0810 23:32:44.633412 (+ 2216us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.633431 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:44.633573 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:44.633980 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:44.633984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:44.634013 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:44.634358 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:44.634387 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.635897 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.635903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:44.638341 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:44.638356 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.641265 (+ 2909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.641269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:44.641448 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.641482 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:32:44.642319 (+ 837us) client_negotiation.cc:240] Negotiation successful
0810 23:32:44.642462 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2114,"thread_start_us":1929,"threads_started":1}
I20260810 23:32:44.643625 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.628838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46437 (local address 127.0.0.1:47810)
0810 23:32:44.628997 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.629019 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:44.629113 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:44.637311 (+ 8198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:44.637342 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:44.637371 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:44.637819 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:44.637831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.639311 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.639318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:44.641273 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:44.641288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.642895 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.642902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:44.643075 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.643111 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:32:44.643218 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 23:32:44.643364 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:32:44.643617 21390 tablet_service.cc:1467] Processing CreateTablet for tablet e790a08c8f484746b6b8fd51e682ac35 (DEFAULT_TABLE table=test-table [id=47aa5a19dde9410ea0cce9e687675355]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:44.643733 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.628838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47810 (local address 127.9.48.193:46437)
0810 23:32:44.636904 (+ 8066us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:44.636911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:44.636930 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:32:44.636998 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:44.637004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:44.637096 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:44.637970 (+ 874us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:44.638144 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.639189 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.641455 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.642649 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.642790 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.642930 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:44.643244 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 23:32:44.643396 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7926,"thread_start_us":7816,"threads_started":1}
I20260810 23:32:44.644048 21390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e790a08c8f484746b6b8fd51e682ac35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:44.646029 21632 tablet_bootstrap.cc:492] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90: Bootstrap starting.
I20260810 23:32:44.646284 20942 tablet_service.cc:1467] Processing CreateTablet for tablet e790a08c8f484746b6b8fd51e682ac35 (DEFAULT_TABLE table=test-table [id=47aa5a19dde9410ea0cce9e687675355]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:44.646719 20942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e790a08c8f484746b6b8fd51e682ac35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:44.647841 21632 tablet_bootstrap.cc:654] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:44.649623 21634 tablet_bootstrap.cc:492] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847: Bootstrap starting.
I20260810 23:32:44.651280 21634 tablet_bootstrap.cc:654] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:44.653456 21634 tablet_bootstrap.cc:492] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847: No bootstrap required, opened a new log
I20260810 23:32:44.653621 21634 ts_tablet_manager.cc:1397] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 23:32:44.654260 21634 raft_consensus.cc:348] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } }
I20260810 23:32:44.654484 21634 raft_consensus.cc:374] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:44.654575 21634 raft_consensus.cc:729] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47c185c80b264d0593a81622c4ef6847, State: Initialized, Role: FOLLOWER
I20260810 23:32:44.654870 21634 consensus_queue.cc:260] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } }
I20260810 23:32:44.655671 21634 ts_tablet_manager.cc:1428] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:32:44.657390 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.656379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.195:48215)
0810 23:32:44.656957 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.657118 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"threads_started":1}
I20260810 23:32:44.659494 21485 heartbeater.cc:503] Master 127.9.48.252:34131 was elected leader, sending a full tablet report...
I20260810 23:32:44.660509 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.660170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.195:57547)
0810 23:32:44.660352 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.660412 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260810 23:32:44.661757 21632 tablet_bootstrap.cc:492] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90: No bootstrap required, opened a new log
I20260810 23:32:44.661919 21632 ts_tablet_manager.cc:1397] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260810 23:32:44.662108 21637 tablet_bootstrap.cc:492] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224: Bootstrap starting.
I20260810 23:32:44.662734 21632 raft_consensus.cc:348] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } }
I20260810 23:32:44.663023 21632 raft_consensus.cc:374] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:44.663132 21632 raft_consensus.cc:729] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7651a6bfd3488d88b587ed9db9ee90, State: Initialized, Role: FOLLOWER
I20260810 23:32:44.663399 21632 consensus_queue.cc:260] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } }
I20260810 23:32:44.664213 21637 tablet_bootstrap.cc:654] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:44.664280 21632 ts_tablet_manager.cc:1428] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 23:32:44.666610 21271 heartbeater.cc:503] Master 127.9.48.252:34131 was elected leader, sending a full tablet report...
I20260810 23:32:44.670290 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.664953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.194:43449)
0810 23:32:44.669734 (+ 4781us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.669912 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4587,"thread_start_us":4437,"threads_started":1}
I20260810 23:32:44.671293 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.670939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.194:51095)
0810 23:32:44.671090 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.671165 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 23:32:44.671635 21637 tablet_bootstrap.cc:492] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224: No bootstrap required, opened a new log
I20260810 23:32:44.671797 21637 ts_tablet_manager.cc:1397] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260810 23:32:44.672452 21637 raft_consensus.cc:348] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } }
I20260810 23:32:44.672724 21637 raft_consensus.cc:374] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:44.672856 21637 raft_consensus.cc:729] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94048588350439e91217f75d0058224, State: Initialized, Role: FOLLOWER
I20260810 23:32:44.673074 21637 consensus_queue.cc:260] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } }
I20260810 23:32:44.673732 21641 raft_consensus.cc:482] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:44.673832 21637 ts_tablet_manager.cc:1428] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 23:32:44.673921 21641 raft_consensus.cc:504] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } }
I20260810 23:32:44.676072 21050 heartbeater.cc:503] Master 127.9.48.252:34131 was elected leader, sending a full tablet report...
I20260810 23:32:44.676584 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.674115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.193:42399)
0810 23:32:44.676039 (+ 1924us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.676238 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1790,"thread_start_us":1650,"threads_started":1}
I20260810 23:32:44.680987 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.678182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.193:40211)
0810 23:32:44.680740 (+ 2558us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.680834 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2457,"threads_started":1}
I20260810 23:32:44.682591 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.676029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:44787 (local address 127.9.48.193:46437)
0810 23:32:44.676218 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:44.676225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:44.676252 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:32:44.676549 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:44.676561 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:44.676643 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:44.676746 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:44.677395 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.678401 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.680341 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.681691 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.681876 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.681952 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:44.682234 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 23:32:44.682362 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:32:44.682611 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.676037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46437 (local address 127.9.48.194:44787)
0810 23:32:44.676181 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.676199 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:44.676342 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:44.676915 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:44.676921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:44.676948 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:44.677218 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:44.677231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.678778 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.678784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:44.680182 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:44.680195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.681878 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.681885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:44.682065 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.682090 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:32:44.682196 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 23:32:44.682342 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 23:32:44.683492 21641 leader_election.cc:288] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c94048588350439e91217f75d0058224 (127.9.48.193:46437), 47c185c80b264d0593a81622c4ef6847 (127.9.48.195:41079)
I20260810 23:32:44.683769 20979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e790a08c8f484746b6b8fd51e682ac35" candidate_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94048588350439e91217f75d0058224" is_pre_election: true
I20260810 23:32:44.684160 20979 raft_consensus.cc:2454] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d7651a6bfd3488d88b587ed9db9ee90 in term 0.
I20260810 23:32:44.684894 21106 leader_election.cc:302] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d7651a6bfd3488d88b587ed9db9ee90, c94048588350439e91217f75d0058224; no voters:
I20260810 23:32:44.685278 21641 raft_consensus.cc:2781] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:44.685441 21641 raft_consensus.cc:482] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:44.685576 21641 raft_consensus.cc:3037] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:44.688576 21641 raft_consensus.cc:504] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } }
I20260810 23:32:44.689244 21641 leader_election.cc:288] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [CANDIDATE]: Term 1 election: Requested vote from peers c94048588350439e91217f75d0058224 (127.9.48.193:46437), 47c185c80b264d0593a81622c4ef6847 (127.9.48.195:41079)
I20260810 23:32:44.689802 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.678246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:55009 (local address 127.9.48.195:41079)
0810 23:32:44.678700 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:44.678707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:44.682909 (+ 4202us) server_negotiation.cc:407] Connection header received
0810 23:32:44.683061 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:44.683069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:44.683152 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:44.683234 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:44.683637 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.685100 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.687525 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.688898 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.689019 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.689070 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:44.689515 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 23:32:44.689611 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260810 23:32:44.689836 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.677808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:41079 (local address 127.9.48.194:55009)
0810 23:32:44.682806 (+ 4998us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.682826 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:44.682950 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:44.683245 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:44.683250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:44.683273 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:44.683470 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:44.683486 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.685268 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.685276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:44.687320 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:44.687341 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.689087 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.689092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:44.689327 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.689363 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:32:44.689460 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:32:44.689574 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4908,"thread_start_us":10602,"threads_started":1}
I20260810 23:32:44.690539 20979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e790a08c8f484746b6b8fd51e682ac35" candidate_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94048588350439e91217f75d0058224"
I20260810 23:32:44.690769 20979 raft_consensus.cc:3037] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:44.690752 21417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e790a08c8f484746b6b8fd51e682ac35" candidate_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c185c80b264d0593a81622c4ef6847" is_pre_election: true
I20260810 23:32:44.691018 21417 raft_consensus.cc:2454] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d7651a6bfd3488d88b587ed9db9ee90 in term 0.
I20260810 23:32:44.690960 21416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e790a08c8f484746b6b8fd51e682ac35" candidate_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c185c80b264d0593a81622c4ef6847"
I20260810 23:32:44.691275 21416 raft_consensus.cc:3037] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:44.693228 20979 raft_consensus.cc:2454] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d7651a6bfd3488d88b587ed9db9ee90 in term 1.
I20260810 23:32:44.693601 21106 leader_election.cc:302] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d7651a6bfd3488d88b587ed9db9ee90, c94048588350439e91217f75d0058224; no voters:
I20260810 23:32:44.693723 21416 raft_consensus.cc:2454] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d7651a6bfd3488d88b587ed9db9ee90 in term 1.
I20260810 23:32:44.693890 21641 raft_consensus.cc:2781] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:44.694469 21641 raft_consensus.cc:686] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 1 LEADER]: Becoming Leader. State: Replica: 2d7651a6bfd3488d88b587ed9db9ee90, State: Running, Role: LEADER
I20260810 23:32:44.694913 21641 consensus_queue.cc:237] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } } peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } }
I20260810 23:32:44.695411 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.694994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.194:40393)
0810 23:32:44.695146 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.695265 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 23:32:44.698393 20761 catalog_manager.cc:5495] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d7651a6bfd3488d88b587ed9db9ee90 (127.9.48.194). New cstate: current_term: 1 leader_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d7651a6bfd3488d88b587ed9db9ee90" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 } health_report { overall_health: UNKNOWN } } }
I20260810 23:32:44.721210 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.714951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.9.48.194:43199)
0810 23:32:44.715143 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:44.715149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:44.715230 (+ 81us) server_negotiation.cc:407] Connection header received
0810 23:32:44.715419 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:44.715423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:44.715503 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:44.715638 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:32:44.716060 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.716897 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.718792 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.719301 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.719423 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.719934 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:32:44.720508 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:32:44.720714 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:44.720840 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 23:32:44.720909 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:32:44.725689 21416 raft_consensus.cc:1261] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847 [term 1 FOLLOWER]: Refusing update from remote peer 2d7651a6bfd3488d88b587ed9db9ee90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:44.726054 20979 raft_consensus.cc:1261] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224 [term 1 FOLLOWER]: Refusing update from remote peer 2d7651a6bfd3488d88b587ed9db9ee90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:44.726519 21653 consensus_queue.cc:1035] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47c185c80b264d0593a81622c4ef6847" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 41079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:44.726867 21641 consensus_queue.cc:1035] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c94048588350439e91217f75d0058224" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:44.729475 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.728956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.195:47381)
0810 23:32:44.729156 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.729262 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 23:32:44.730386 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.729688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.193:47935)
0810 23:32:44.729963 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.730154 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260810 23:32:44.735584 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.734567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.195:34483)
0810 23:32:44.735336 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.735427 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":660}
I20260810 23:32:44.735591 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.735121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.194:57627)
0810 23:32:44.735343 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.735444 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260810 23:32:44.759449 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.758672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.194:40775)
0810 23:32:44.759107 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.759222 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308}
I20260810 23:32:44.760933 20761 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:44.761294 20761 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:44.763530 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.763007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.9.48.193:35427)
0810 23:32:44.763271 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:44.763363 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260810 23:32:44.768365 20761 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a4caf350fc747079975acc1aceb6a3c: Sending DeleteTablet for 3 replicas of tablet e790a08c8f484746b6b8fd51e682ac35
I20260810 23:32:44.769547 21168 tablet_service.cc:1514] Processing DeleteTablet for tablet e790a08c8f484746b6b8fd51e682ac35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:44 UTC) from {username='oryx', principal='oryx/127.9.48.252@KRBTEST.COM'} at 127.0.0.1:44742
I20260810 23:32:44.769657 20942 tablet_service.cc:1514] Processing DeleteTablet for tablet e790a08c8f484746b6b8fd51e682ac35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:44 UTC) from {username='oryx', principal='oryx/127.9.48.252@KRBTEST.COM'} at 127.0.0.1:47810
I20260810 23:32:44.769724 20940 tablet_service.cc:1514] Processing DeleteTablet for tablet e790a08c8f484746b6b8fd51e682ac35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:44 UTC) from {username='oryx', principal='oryx/127.9.48.252@KRBTEST.COM'} at 127.0.0.1:47810
I20260810 23:32:44.770172 21390 tablet_service.cc:1514] Processing DeleteTablet for tablet e790a08c8f484746b6b8fd51e682ac35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:44 UTC) from {username='oryx', principal='oryx/127.9.48.252@KRBTEST.COM'} at 127.0.0.1:57484
I20260810 23:32:44.770952 21676 tablet_replica.cc:330] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847: stopping tablet replica
I20260810 23:32:44.771189 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 20740
I20260810 23:32:44.771425 21676 raft_consensus.cc:2229] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:44.771829 21676 raft_consensus.cc:2258] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:44.772938 21675 tablet_replica.cc:330] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224: stopping tablet replica
I20260810 23:32:44.773119 21675 raft_consensus.cc:2229] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:44.773435 21675 raft_consensus.cc:2258] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:44.774012 21673 tablet_replica.cc:330] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90: stopping tablet replica
I20260810 23:32:44.774189 21673 raft_consensus.cc:2229] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:44.774425 21675 ts_tablet_manager.cc:1905] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:44.774626 21676 ts_tablet_manager.cc:1905] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:44.774765 21673 raft_consensus.cc:2258] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:44.775832 21673 ts_tablet_manager.cc:1905] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:44.779731 21675 ts_tablet_manager.cc:1918] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:44.779865 21675 log.cc:1196] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/e790a08c8f484746b6b8fd51e682ac35
I20260810 23:32:44.780346 21675 ts_tablet_manager.cc:1939] T e790a08c8f484746b6b8fd51e682ac35 P c94048588350439e91217f75d0058224: Deleting consensus metadata
I20260810 23:32:44.781478 20733 catalog_manager.cc:4841] TS c94048588350439e91217f75d0058224 (127.9.48.193:46437): tablet e790a08c8f484746b6b8fd51e682ac35 (table test-table [id=47aa5a19dde9410ea0cce9e687675355]) successfully deleted
W20260810 23:32:44.782327 20733 catalog_manager.cc:4808] TS c94048588350439e91217f75d0058224 (127.9.48.193:46437): delete failed for tablet e790a08c8f484746b6b8fd51e682ac35 because the tablet was not found. No further retry: Not found: Tablet not found: e790a08c8f484746b6b8fd51e682ac35
I20260810 23:32:44.782514 21673 ts_tablet_manager.cc:1918] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:44.782655 21673 log.cc:1196] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/e790a08c8f484746b6b8fd51e682ac35
I20260810 23:32:44.783118 21673 ts_tablet_manager.cc:1939] T e790a08c8f484746b6b8fd51e682ac35 P 2d7651a6bfd3488d88b587ed9db9ee90: Deleting consensus metadata
I20260810 23:32:44.784169 21676 ts_tablet_manager.cc:1918] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:44.784183 20732 catalog_manager.cc:4841] TS 2d7651a6bfd3488d88b587ed9db9ee90 (127.9.48.194:43199): tablet e790a08c8f484746b6b8fd51e682ac35 (table test-table [id=47aa5a19dde9410ea0cce9e687675355]) successfully deleted
I20260810 23:32:44.784332 21676 log.cc:1196] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/e790a08c8f484746b6b8fd51e682ac35
I20260810 23:32:44.784797 21676 ts_tablet_manager.cc:1939] T e790a08c8f484746b6b8fd51e682ac35 P 47c185c80b264d0593a81622c4ef6847: Deleting consensus metadata
I20260810 23:32:44.785768 20738 catalog_manager.cc:4841] TS 47c185c80b264d0593a81622c4ef6847 (127.9.48.195:41079): tablet e790a08c8f484746b6b8fd51e682ac35 (table test-table [id=47aa5a19dde9410ea0cce9e687675355]) successfully deleted
I20260810 23:32:44.791647 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.772269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.9.48.193:46437)
0810 23:32:44.773052 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:44.773058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:44.773077 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:32:44.773142 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:44.773150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:44.773232 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:44.773360 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:44.778302 (+ 4942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.779429 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.781402 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.782034 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.782171 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.785290 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:44.785303 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:44.785318 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:44.785349 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:44.788449 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:44.789297 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:44.789305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:44.789307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:44.789424 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:44.789882 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:44.789888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:44.789891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:44.790409 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:44.790753 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:44.791026 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 23:32:44.791266 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675}
I20260810 23:32:44.792337 21040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:44.921674 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 20740
I20260810 23:32:44.952826 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 21070
I20260810 23:32:44.969146 21626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:44.953924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44760 (local address 127.9.48.194:43199)
0810 23:32:44.954143 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:44.954149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:44.954171 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:32:44.954249 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:44.954255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:44.954346 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:44.954469 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:44.955732 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.956961 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.958210 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:44.958883 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:44.959015 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:44.961921 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:44.961943 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:44.961963 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:44.962002 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:44.966224 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:44.966943 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:44.966954 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:44.966958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:44.967170 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:44.967536 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:44.967546 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:44.967550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:44.967983 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:44.968317 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:44.968483 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 23:32:44.968729 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:32:44.969894 21264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:45.089903 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 21070
I20260810 23:32:45.120151 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 21296
I20260810 23:32:45.134327 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:45.120951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.9.48.195:41079)
0810 23:32:45.121178 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:45.121183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:45.121222 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:32:45.121395 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:45.121401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:45.121482 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:45.121640 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:45.123080 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:45.123815 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:45.125345 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:45.125796 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:45.125940 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:45.128621 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:45.128635 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:45.128652 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:45.128698 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:45.131568 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:45.132197 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:45.132204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:45.132207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:45.132353 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:45.132859 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:45.132864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:45.132867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:45.133283 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:45.133512 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:45.133807 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 23:32:45.133975 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:32:45.134982 21480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:45.178100 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:45.177409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:33418)
0810 23:32:45.177835 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:45.177924 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":222,"threads_started":1}
I20260810 23:32:45.179386 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:45.179057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40553 (local address 127.0.0.1:33420)
0810 23:32:45.179202 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:45.179272 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 23:32:45.230055 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 21296
I20260810 23:32:45.260959 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 20222
I20260810 23:32:45.262032 20515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:45.381685 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 20222
I20260810 23:32:45.414184 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 20400
I20260810 23:32:45.415352 20815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:45.539141 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 20400
2026-08-10T23:32:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10357 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:32:45 dist-test-slave-04bb krb5kdc[21701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:32:45 dist-test-slave-04bb krb5kdc[21701](info): set up 2 sockets
Aug 10 23:32:45 dist-test-slave-04bb krb5kdc[21701](info): commencing operation
krb5kdc: starting...
W20260810 23:32:46.888751 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.295s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:32:47 dist-test-slave-04bb krb5kdc[21701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:32:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.254.keytab.
Entry for principal oryx/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.254.keytab.
I20260810 23:32:47.148710 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:42567
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.254.keytab
--principal=oryx/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:42567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:47.480499 21719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:47.480887 21719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:47.481058 21719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:47.488044 21719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:47.488178 21719 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:32:47.488296 21719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:47.488401 21719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:47.488495 21719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:47.488601 21719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:47.498169 21719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:42567
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.254.keytab
--principal=oryx/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:42567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:47.499464 21719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:47.500836 21719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:47.512934 21724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:47.512853 21725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:47.512902 21728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:47.513850 21719 server_base.cc:1034] running on GCE node
I20260810 23:32:47.514602 21719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:47.515928 21719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:47.517181 21719 hybrid_clock.cc:648] HybridClock initialized: now 1786404767517117 us; error 39 us; skew 500 ppm
I20260810 23:32:47.517511 21719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:47 dist-test-slave-04bb krb5kdc[21701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404767, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:47.520793 21719 init.cc:377] Logged in from keytab as oryx/127.9.48.254@KRBTEST.COM (short username oryx)
I20260810 23:32:47.522251 21719 webserver.cc:460] Webserver started at http://127.9.48.254:41477/ using document root <none> and password file <none>
I20260810 23:32:47.522929 21719 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:47.523057 21719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:47.523403 21719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:47.526371 21719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "1e72b0b8f35d4432ba3391ead7780184"
format_stamp: "Formatted at 2026-08-10 23:32:47 on dist-test-slave-04bb"
server_key: "426a7a63d1160571d8fe4512449aa868"
server_key_iv: "c9ae1683a6928ee8c65fb1cc7200a015"
server_key_version: "encryptionkey@0"
I20260810 23:32:47.527173 21719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "1e72b0b8f35d4432ba3391ead7780184"
format_stamp: "Formatted at 2026-08-10 23:32:47 on dist-test-slave-04bb"
server_key: "426a7a63d1160571d8fe4512449aa868"
server_key_iv: "c9ae1683a6928ee8c65fb1cc7200a015"
server_key_version: "encryptionkey@0"
I20260810 23:32:47.532326 21719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 23:32:47.535815 21736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:47.537238 21719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:47.537434 21719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "1e72b0b8f35d4432ba3391ead7780184"
format_stamp: "Formatted at 2026-08-10 23:32:47 on dist-test-slave-04bb"
server_key: "426a7a63d1160571d8fe4512449aa868"
server_key_iv: "c9ae1683a6928ee8c65fb1cc7200a015"
server_key_version: "encryptionkey@0"
I20260810 23:32:47.537652 21719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:47.550750 21719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:47.555219 21719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:47.555500 21719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:47.578019 21719 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:42567
I20260810 23:32:47.578027 21834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:42567 every 8 connection(s)
I20260810 23:32:47.579418 21719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:47.583142 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 21719
I20260810 23:32:47.583389 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:32:47.583863 21835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:47.584360 9411 external_mini_cluster.cc:1428] Setting key 68405049fb3c2f5bf2d46f386eb08242
I20260810 23:32:47.599804 21835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184: Bootstrap starting.
Aug 10 23:32:47 dist-test-slave-04bb krb5kdc[21701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
I20260810 23:32:47.603681 21835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:47.604959 21835 log.cc:824] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:47.608456 21835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184: No bootstrap required, opened a new log
I20260810 23:32:47.609400 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:47.586622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33142 (local address 127.9.48.254:42567)
0810 23:32:47.587566 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:47.587578 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:47.587627 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:32:47.589343 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:47.589365 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:47.589742 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:47.590073 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:47.592187 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:47.593627 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:47.595584 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:47.596347 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:47.596489 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:47.601701 (+ 5212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:47.601721 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:47.601735 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:47.601772 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:47.605040 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:47.605956 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:47.605968 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:47.605978 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:47.606214 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:47.606714 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:47.606718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:47.606720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:47.607230 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:47.607437 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:47.607961 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 23:32:47.608297 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":487,"threads_started":1}
I20260810 23:32:47.620777 21835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e72b0b8f35d4432ba3391ead7780184" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 42567 } }
I20260810 23:32:47.621498 21835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:47.621718 21835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e72b0b8f35d4432ba3391ead7780184, State: Initialized, Role: FOLLOWER
I20260810 23:32:47.622658 21835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e72b0b8f35d4432ba3391ead7780184" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 42567 } }
I20260810 23:32:47.623096 21835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:32:47.623317 21835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:32:47.623612 21835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:47.627234 21835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e72b0b8f35d4432ba3391ead7780184" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 42567 } }
I20260810 23:32:47.627868 21835 leader_election.cc:302] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e72b0b8f35d4432ba3391ead7780184; no voters:
I20260810 23:32:47.628566 21835 leader_election.cc:288] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:32:47.628703 21844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:47.629565 21844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [term 1 LEADER]: Becoming Leader. State: Replica: 1e72b0b8f35d4432ba3391ead7780184, State: Running, Role: LEADER
I20260810 23:32:47.630177 21844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e72b0b8f35d4432ba3391ead7780184" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 42567 } }
I20260810 23:32:47.631273 21835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:47.634083 21846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e72b0b8f35d4432ba3391ead7780184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e72b0b8f35d4432ba3391ead7780184" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 42567 } } }
I20260810 23:32:47.634037 21847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e72b0b8f35d4432ba3391ead7780184. Latest consensus state: current_term: 1 leader_uuid: "1e72b0b8f35d4432ba3391ead7780184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e72b0b8f35d4432ba3391ead7780184" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 42567 } } }
I20260810 23:32:47.634599 21847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:47.634598 21846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:47.643544 21857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:47.649369 21857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:47.671368 21857 catalog_manager.cc:1347] Generated new cluster ID: ab13ceb2d6ec46118fdf86ee45d097f2
I20260810 23:32:47.671514 21857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:47.693356 21857 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:32:47.695200 21857 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:47.703608 21857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184: Generated new TSK 0
I20260810 23:32:47.704387 21857 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.193.keytab.
Entry for principal oryx/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.193.keytab.
I20260810 23:32:47.769033 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:42567
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.193.keytab
--principal=oryx/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:48.113826 21877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:48.114095 21877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:48.114251 21877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:48.121435 21877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:48.121536 21877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:48.121706 21877 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:32:48.121840 21877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:32:48.131927 21877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.193.keytab
--principal=oryx/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.48.254:42567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:48.133350 21877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:48.134850 21877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:48.148720 21883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:48.148738 21884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:48.148698 21887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:48.149575 21877 server_base.cc:1034] running on GCE node
I20260810 23:32:48.150197 21877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:48.151096 21877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:48.152364 21877 hybrid_clock.cc:648] HybridClock initialized: now 1786404768152269 us; error 50 us; skew 500 ppm
I20260810 23:32:48.152721 21877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:48 dist-test-slave-04bb krb5kdc[21701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404768, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:48.160681 21877 init.cc:377] Logged in from keytab as oryx/127.9.48.193@KRBTEST.COM (short username oryx)
I20260810 23:32:48.162132 21877 webserver.cc:460] Webserver started at http://127.9.48.193:37831/ using document root <none> and password file <none>
I20260810 23:32:48.162834 21877 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:48.162968 21877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:48.163308 21877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:48.166401 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "c4418ca816004f7a8734905d079185fb"
format_stamp: "Formatted at 2026-08-10 23:32:48 on dist-test-slave-04bb"
server_key: "05079127c8f698e200e9cdd040353613"
server_key_iv: "d302208c2061eb57d1f93e4c35d4f7b9"
server_key_version: "encryptionkey@0"
I20260810 23:32:48.167217 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "c4418ca816004f7a8734905d079185fb"
format_stamp: "Formatted at 2026-08-10 23:32:48 on dist-test-slave-04bb"
server_key: "05079127c8f698e200e9cdd040353613"
server_key_iv: "d302208c2061eb57d1f93e4c35d4f7b9"
server_key_version: "encryptionkey@0"
I20260810 23:32:48.172369 21877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 23:32:48.175930 21898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:48.177250 21877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:48.177428 21877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "c4418ca816004f7a8734905d079185fb"
format_stamp: "Formatted at 2026-08-10 23:32:48 on dist-test-slave-04bb"
server_key: "05079127c8f698e200e9cdd040353613"
server_key_iv: "d302208c2061eb57d1f93e4c35d4f7b9"
server_key_version: "encryptionkey@0"
I20260810 23:32:48.177675 21877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:48.197820 21877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:48.201988 21877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:48.202275 21877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:48.205303 21877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:48.205416 21877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:48.205663 21877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:48.205766 21877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:32:48 dist-test-slave-04bb krb5kdc[21701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404768, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
I20260810 23:32:48.228821 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:48.205719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:56969 (local address 127.9.48.254:42567)
0810 23:32:48.206242 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:48.206248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:48.207515 (+ 1267us) server_negotiation.cc:407] Connection header received
0810 23:32:48.209830 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:48.209836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:48.209933 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:48.210053 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:48.212349 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.213427 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.216735 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.217286 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.217449 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:48.222332 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:48.222337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:48.222341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:48.222376 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:48.225411 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:48.226408 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:48.226414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:48.226419 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:48.226545 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:48.226996 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:48.227000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:48.227003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:48.227268 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:48.227429 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:48.228166 (+ 737us) server_negotiation.cc:299] Negotiation successful
0810 23:32:48.228389 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":264,"threads_started":1}
I20260810 23:32:48.229905 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:48.206405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:42567 (local address 127.9.48.193:56969)
0810 23:32:48.207312 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:48.207376 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:48.209551 (+ 2175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:48.210354 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:48.210371 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:48.211193 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:48.212158 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:48.212185 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.213627 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.213637 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:48.216505 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:48.216550 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.217604 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.217616 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:48.218050 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:48.219006 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:48.219057 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:48.222103 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:48.225562 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:48.225571 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:48.225587 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:48.226260 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:48.226667 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:48.226672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:48.226676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:48.226863 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:48.227404 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:48.227412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:48.227770 (+ 358us) client_negotiation.cc:769] Sending connection context
0810 23:32:48.228051 (+ 281us) client_negotiation.cc:240] Negotiation successful
0810 23:32:48.228613 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":387,"threads_started":1}
I20260810 23:32:48.252869 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:48.237078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:41447 (local address 127.9.48.254:42567)
0810 23:32:48.237328 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:48.237337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:48.237940 (+ 603us) server_negotiation.cc:407] Connection header received
0810 23:32:48.238135 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:48.238139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:48.238199 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:48.238277 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:48.239863 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.240693 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.243100 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.243620 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.243762 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:48.246137 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:48.246142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:48.246145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:48.246175 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:48.249625 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:48.250739 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:48.250747 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:48.250753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:48.250929 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:48.251481 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:48.251487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:48.251490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:48.251769 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:48.251900 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:48.252391 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 23:32:48.252618 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 23:32:48.253338 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:48.237017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:42567 (local address 127.9.48.193:41447)
0810 23:32:48.237810 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:48.237837 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:48.238009 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:48.238347 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:48.238355 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:48.239213 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:48.239678 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:48.239694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.240836 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.240844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:48.242963 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:48.242983 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.243850 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.243860 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:48.244141 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:48.244924 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:48.244942 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:48.245936 (+ 994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:48.249801 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:48.249806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:48.249811 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:48.250600 (+ 789us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:48.251068 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:48.251076 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:48.251082 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:48.251297 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:48.251890 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:48.251895 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:48.252108 (+ 213us) client_negotiation.cc:769] Sending connection context
0810 23:32:48.252318 (+ 210us) client_negotiation.cc:240] Negotiation successful
0810 23:32:48.252712 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":429,"threads_started":1}
I20260810 23:32:48.262717 21877 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:40931
I20260810 23:32:48.262763 22075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:40931 every 8 connection(s)
I20260810 23:32:48.263960 21877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:32:48.266095 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 21877
I20260810 23:32:48.266108 22078 heartbeater.cc:347] Connected to a master server at 127.9.48.254:42567
I20260810 23:32:48.266315 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:32:48.266355 22078 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:48.266968 9411 external_mini_cluster.cc:1428] Setting key 2f2dbb0de2dcb2c82ac3e7fa6a1f1c39
I20260810 23:32:48.266983 22078 heartbeater.cc:511] Master 127.9.48.254:42567 requested a full tablet report, sending...
I20260810 23:32:48.269320 21764 ts_manager.cc:194] Registered new tserver with Master: c4418ca816004f7a8734905d079185fb (127.9.48.193:40931)
I20260810 23:32:48.271561 21764 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:56969
WARNING: no policy specified for oryx/127.9.48.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.194.keytab.
Entry for principal oryx/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.194.keytab.
I20260810 23:32:48.325565 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:42567
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.194.keytab
--principal=oryx/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:48.662762 22083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:48.663084 22083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:48.663275 22083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:48.670563 22083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:48.670706 22083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:48.670861 22083 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:32:48.671000 22083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:32:48.681056 22083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.194.keytab
--principal=oryx/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.48.254:42567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:48.682466 22083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:48.683821 22083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:48.697332 22092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:48.697455 22100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:48.697497 22098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:48.698506 22083 server_base.cc:1034] running on GCE node
I20260810 23:32:48.699057 22083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:48.699982 22083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:48.701251 22083 hybrid_clock.cc:648] HybridClock initialized: now 1786404768701178 us; error 38 us; skew 500 ppm
I20260810 23:32:48.701557 22083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:48 dist-test-slave-04bb krb5kdc[21701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404768, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:48.705459 22083 init.cc:377] Logged in from keytab as oryx/127.9.48.194@KRBTEST.COM (short username oryx)
I20260810 23:32:48.706899 22083 webserver.cc:460] Webserver started at http://127.9.48.194:40197/ using document root <none> and password file <none>
I20260810 23:32:48.707605 22083 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:48.707741 22083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:48.708097 22083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:48.711194 22083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "8049a54497984825b0a1d9a4c5494ffb"
format_stamp: "Formatted at 2026-08-10 23:32:48 on dist-test-slave-04bb"
server_key: "885543c024c976cd9b70d67e05ad7faa"
server_key_iv: "ece0d599f6375f39596f10f0f9bd7d7c"
server_key_version: "encryptionkey@0"
I20260810 23:32:48.712102 22083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "8049a54497984825b0a1d9a4c5494ffb"
format_stamp: "Formatted at 2026-08-10 23:32:48 on dist-test-slave-04bb"
server_key: "885543c024c976cd9b70d67e05ad7faa"
server_key_iv: "ece0d599f6375f39596f10f0f9bd7d7c"
server_key_version: "encryptionkey@0"
I20260810 23:32:48.717406 22083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:32:48.720948 22108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:48.722259 22083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:32:48.722436 22083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "8049a54497984825b0a1d9a4c5494ffb"
format_stamp: "Formatted at 2026-08-10 23:32:48 on dist-test-slave-04bb"
server_key: "885543c024c976cd9b70d67e05ad7faa"
server_key_iv: "ece0d599f6375f39596f10f0f9bd7d7c"
server_key_version: "encryptionkey@0"
I20260810 23:32:48.722659 22083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:48.745484 22083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:48.750042 22083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:48.750312 22083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:48.753070 22083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:48.753252 22083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 23:32:48.753453 22083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:48.753589 22083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:32:48 dist-test-slave-04bb krb5kdc[21701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404768, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
I20260810 23:32:48.777483 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:48.753737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:38077 (local address 127.9.48.254:42567)
0810 23:32:48.756911 (+ 3174us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:48.756918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:48.756941 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:32:48.757767 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:48.757776 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:48.757867 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:48.757995 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:48.759717 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.761252 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.764008 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.764532 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.764699 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:48.769652 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:48.769662 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:48.769670 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:48.769727 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:48.773874 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:48.774940 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:48.774946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:48.774952 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:48.775104 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:48.775567 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:48.775573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:48.775576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:48.775873 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:48.776046 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:48.776825 (+ 779us) server_negotiation.cc:299] Negotiation successful
0810 23:32:48.777068 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3043,"thread_start_us":2933,"threads_started":1}
I20260810 23:32:48.778502 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:48.754408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:42567 (local address 127.9.48.194:38077)
0810 23:32:48.755275 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:48.755320 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:48.757460 (+ 2140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:48.758244 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:48.758251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:48.758768 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:48.759441 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:48.759487 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.761439 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.761447 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:48.763833 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:48.763845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.764888 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.764899 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:48.765283 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:48.766254 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:48.766311 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:48.769375 (+ 3064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:48.774034 (+ 4659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:48.774050 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:48.774070 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:48.774803 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:48.775240 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:48.775244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:48.775246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:48.775446 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:48.776020 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:48.776027 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:48.776401 (+ 374us) client_negotiation.cc:769] Sending connection context
0810 23:32:48.776740 (+ 339us) client_negotiation.cc:240] Negotiation successful
0810 23:32:48.777311 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":458,"threads_started":1}
I20260810 23:32:48.800171 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:48.784779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:55549 (local address 127.9.48.254:42567)
0810 23:32:48.784995 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:48.785001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:48.785663 (+ 662us) server_negotiation.cc:407] Connection header received
0810 23:32:48.785868 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:48.785874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:48.785976 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:48.786082 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:48.787623 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.788742 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.790780 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.791189 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.791330 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:48.793594 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:48.793600 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:48.793603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:48.793635 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:48.797073 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:48.798109 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:48.798116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:48.798119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:48.798246 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:48.798631 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:48.798637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:48.798640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:48.798971 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:48.799154 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:48.799576 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 23:32:48.799818 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 23:32:48.800263 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:48.784863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:42567 (local address 127.9.48.194:55549)
0810 23:32:48.785557 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:48.785573 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:48.785737 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:48.786291 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:48.786296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:48.786913 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:48.787463 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:48.787478 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.788889 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.788894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:48.790571 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:48.790597 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:48.791392 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:48.791400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:48.791586 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:48.792289 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:48.792314 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:48.793397 (+ 1083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:48.797259 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:48.797265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:48.797269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:48.797984 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:48.798368 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:48.798373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:48.798376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:48.798513 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:48.799115 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:48.799120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:48.799318 (+ 198us) client_negotiation.cc:769] Sending connection context
0810 23:32:48.799471 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 23:32:48.799805 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":425,"threads_started":1}
I20260810 23:32:48.810113 22083 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:44465
I20260810 23:32:48.810218 22294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:44465 every 8 connection(s)
I20260810 23:32:48.811494 22083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:32:48.813696 22295 heartbeater.cc:347] Connected to a master server at 127.9.48.254:42567
I20260810 23:32:48.813934 22295 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:48.814589 22295 heartbeater.cc:511] Master 127.9.48.254:42567 requested a full tablet report, sending...
I20260810 23:32:48.815923 21764 ts_manager.cc:194] Registered new tserver with Master: 8049a54497984825b0a1d9a4c5494ffb (127.9.48.194:44465)
I20260810 23:32:48.816829 21764 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:38077
I20260810 23:32:48.821534 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 22083
I20260810 23:32:48.821712 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:32:48.822167 9411 external_mini_cluster.cc:1428] Setting key a27f69ea0ee35ce7b15afc542f875580
WARNING: no policy specified for oryx/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.195.keytab.
Entry for principal oryx/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.195.keytab.
I20260810 23:32:48.884080 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:42567
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.195.keytab
--principal=oryx/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:49.223497 22300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:49.223850 22300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:49.224042 22300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:49.231578 22300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:49.231742 22300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:49.231889 22300 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:32:49.232100 22300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:32:49.242266 22300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.195.keytab
--principal=oryx/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.48.254:42567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:49.243607 22300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:49.244912 22300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:49.258212 22311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:49.258113 22314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:49.258157 22312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:49.259059 22300 server_base.cc:1034] running on GCE node
I20260810 23:32:49.259675 22300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:49.260622 22300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:49.261909 22300 hybrid_clock.cc:648] HybridClock initialized: now 1786404769261829 us; error 61 us; skew 500 ppm
I20260810 23:32:49.262292 22300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:49 dist-test-slave-04bb krb5kdc[21701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:49.265638 22300 init.cc:377] Logged in from keytab as oryx/127.9.48.195@KRBTEST.COM (short username oryx)
I20260810 23:32:49.267258 22300 webserver.cc:460] Webserver started at http://127.9.48.195:40935/ using document root <none> and password file <none>
I20260810 23:32:49.268002 22300 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:49.268118 22300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:49.268460 22300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:49.271440 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "ad8440266c9d4a34b3e431c523f79a71"
format_stamp: "Formatted at 2026-08-10 23:32:49 on dist-test-slave-04bb"
server_key: "69bb7094d851c3c6e768e65db78ba965"
server_key_iv: "40f21de16d06ce0a332806bb8b6f0881"
server_key_version: "encryptionkey@0"
I20260810 23:32:49.272253 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "ad8440266c9d4a34b3e431c523f79a71"
format_stamp: "Formatted at 2026-08-10 23:32:49 on dist-test-slave-04bb"
server_key: "69bb7094d851c3c6e768e65db78ba965"
server_key_iv: "40f21de16d06ce0a332806bb8b6f0881"
server_key_version: "encryptionkey@0"
I20260810 23:32:49.274345 22078 heartbeater.cc:503] Master 127.9.48.254:42567 was elected leader, sending a full tablet report...
I20260810 23:32:49.278002 22300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 23:32:49.282102 22324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:49.283474 22300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:32:49.283764 22300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "ad8440266c9d4a34b3e431c523f79a71"
format_stamp: "Formatted at 2026-08-10 23:32:49 on dist-test-slave-04bb"
server_key: "69bb7094d851c3c6e768e65db78ba965"
server_key_iv: "40f21de16d06ce0a332806bb8b6f0881"
server_key_version: "encryptionkey@0"
I20260810 23:32:49.284036 22300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:49.306908 22300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:49.311681 22300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:49.311959 22300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:49.315074 22300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:49.315173 22300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:49.315372 22300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:49.315435 22300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:32:49 dist-test-slave-04bb krb5kdc[21701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404769, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
I20260810 23:32:49.341718 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.315832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:41293 (local address 127.9.48.254:42567)
0810 23:32:49.316377 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:49.316383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:49.317617 (+ 1234us) server_negotiation.cc:407] Connection header received
0810 23:32:49.320768 (+ 3151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:49.320777 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:49.320895 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:49.320996 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:49.324107 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.325379 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.327688 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.328130 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.328273 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.333779 (+ 5506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:49.333790 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:49.333797 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:49.333851 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:49.337934 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:49.339053 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:49.339060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:49.339066 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:49.339265 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:49.339763 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:49.339768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:49.339772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:49.340059 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:49.340216 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:49.341028 (+ 812us) server_negotiation.cc:299] Negotiation successful
0810 23:32:49.341277 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":289,"threads_started":1}
I20260810 23:32:49.342854 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.316424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:42567 (local address 127.9.48.195:41293)
0810 23:32:49.317403 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:49.317469 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:49.320327 (+ 2858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:49.321721 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:49.321741 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:49.322759 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:49.323849 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.323883 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.325562 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.325567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.327536 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.327549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.328373 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.328382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.328689 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.329420 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:49.329453 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:49.333412 (+ 3959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:49.338113 (+ 4701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:49.338124 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:49.338143 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:49.338896 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:49.339385 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:49.339393 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:49.339400 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:49.339642 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:49.340195 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:49.340202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:49.340619 (+ 417us) client_negotiation.cc:769] Sending connection context
0810 23:32:49.340948 (+ 329us) client_negotiation.cc:240] Negotiation successful
0810 23:32:49.341510 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":341,"threads_started":1}
I20260810 23:32:49.365626 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.350567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:51247 (local address 127.9.48.254:42567)
0810 23:32:49.350765 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:49.350771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:49.350893 (+ 122us) server_negotiation.cc:407] Connection header received
0810 23:32:49.351102 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:49.351108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:49.351178 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:49.351278 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:49.352908 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.353939 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.356095 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.356747 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.356899 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.359550 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:49.359557 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:49.359565 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:49.359610 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:49.362954 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:49.363610 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:49.363619 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:49.363623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:49.363758 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:49.364131 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:49.364137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:49.364142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:49.364418 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:49.364599 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:49.365021 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 23:32:49.365249 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:32:49.365716 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.350116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:42567 (local address 127.9.48.195:51247)
0810 23:32:49.350786 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:49.350809 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:49.350971 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:49.351319 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:49.351326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:49.352096 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:49.352757 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.352770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.354088 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.354093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.355955 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.355977 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.356948 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.356955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.357130 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.357846 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:49.357875 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:49.359322 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:49.363093 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:49.363098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:49.363102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:49.363481 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:49.363881 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:49.363886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:49.363890 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:49.364021 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:49.364566 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:49.364571 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:49.364763 (+ 192us) client_negotiation.cc:769] Sending connection context
0810 23:32:49.364916 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 23:32:49.365247 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":505,"threads_started":1}
I20260810 23:32:49.371881 22300 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:38101
I20260810 23:32:49.371963 22524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:38101 every 8 connection(s)
I20260810 23:32:49.373193 22300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:32:49.375106 22526 heartbeater.cc:347] Connected to a master server at 127.9.48.254:42567
I20260810 23:32:49.375298 22526 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:49.375859 22526 heartbeater.cc:511] Master 127.9.48.254:42567 requested a full tablet report, sending...
I20260810 23:32:49.377131 21764 ts_manager.cc:194] Registered new tserver with Master: ad8440266c9d4a34b3e431c523f79a71 (127.9.48.195:38101)
I20260810 23:32:49.377887 21764 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:41293
I20260810 23:32:49.381817 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 22300
I20260810 23:32:49.382027 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:32:49.382491 9411 external_mini_cluster.cc:1428] Setting key 43915abef27be9eccd42cc779da1834f
I20260810 23:32:49.385411 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:32:49 dist-test-slave-04bb krb5kdc[21701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404769, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:32:49 dist-test-slave-04bb krb5kdc[21701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:32:49 dist-test-slave-04bb krb5kdc[21701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:32:49.448596 22340 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:49.435671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.9.48.254:42567)
0810 23:32:49.435929 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:49.435937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:49.436100 (+ 163us) server_negotiation.cc:407] Connection header received
0810 23:32:49.436381 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:49.436390 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:49.436486 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:49.436640 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:49.438119 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.438991 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.440854 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.441334 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.441488 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.448183 (+ 6695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33148: BlockingRecv error: recv got EOF from 127.0.0.1:33148 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 10 23:32:49 dist-test-slave-04bb krb5kdc[21701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404769, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
I20260810 23:32:49.465641 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.452129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.9.48.254:42567)
0810 23:32:49.452706 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:49.452714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:49.452734 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:32:49.452940 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:49.452946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:49.453021 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:49.453127 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:49.454245 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.454966 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.456195 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.456719 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.456858 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.460447 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:49.460453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:49.460456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:49.460490 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:49.462986 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:49.463637 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:49.463643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:49.463647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:49.463777 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:49.464161 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:49.464167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:49.464170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:49.464445 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:49.464783 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:49.465091 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 23:32:49.465337 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457}
I20260810 23:32:49.469611 21764 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:49.476673 21764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:49.501853 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.489303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:44465 (local address 127.0.0.1:40942)
0810 23:32:49.491448 (+ 2145us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:49.491471 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:49.491603 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:49.493803 (+ 2200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:49.493813 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:49.493871 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:49.494375 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.494392 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.496467 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.496472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.498635 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.498648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.500782 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.500788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.500992 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.501042 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 23:32:49.501170 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 23:32:49.501384 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1979,"mutex_wait_us":81,"thread_start_us":1822,"threads_started":1}
I20260810 23:32:49.501994 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.489271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:38101 (local address 127.0.0.1:46734)
0810 23:32:49.490436 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:49.490473 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:49.490642 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:49.494804 (+ 4162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:49.494811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:49.494850 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:49.495412 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.495430 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.497260 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.497269 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.499354 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.499367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.501145 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.501152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.501348 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.501382 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:32:49.501499 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 23:32:49.501626 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"thread_start_us":5956,"threads_started":1}
I20260810 23:32:49.502398 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.489961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.9.48.195:38101)
0810 23:32:49.494247 (+ 4286us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:49.494253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:49.494278 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:32:49.494362 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:49.494367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:49.494592 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:49.494793 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:49.495606 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.497076 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.499524 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.500909 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.501065 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.501305 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:49.501549 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 23:32:49.501718 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4157,"thread_start_us":2419,"threads_started":1}
I20260810 23:32:49.502794 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.489851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40942 (local address 127.9.48.194:44465)
0810 23:32:49.493249 (+ 3398us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:49.493256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:49.493281 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:32:49.493374 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:49.493380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:49.493564 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:49.493726 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:49.494600 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.496322 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.498797 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.500263 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.500407 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.500646 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:49.502349 (+ 1703us) server_negotiation.cc:299] Negotiation successful
0810 23:32:49.502512 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3274,"thread_start_us":3174,"threads_started":1}
I20260810 23:32:49.504086 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.489639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:40931 (local address 127.0.0.1:39932)
0810 23:32:49.495495 (+ 5856us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:49.495513 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:49.495635 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:49.496441 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:49.496448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:49.496475 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:49.497000 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.497016 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.498950 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.498956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.501010 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.501024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.503319 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.503325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.503494 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.503525 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 23:32:49.503616 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:32:49.503740 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5734,"thread_start_us":534,"threads_started":1}
I20260810 23:32:49.504189 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.490024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.9.48.193:40931)
0810 23:32:49.494046 (+ 4022us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:49.494052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:49.495764 (+ 1712us) server_negotiation.cc:407] Connection header received
0810 23:32:49.495859 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:49.495866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:49.496052 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:49.496211 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:49.497229 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.498812 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.501206 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.502849 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.502993 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.503185 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:49.503739 (+ 554us) server_negotiation.cc:299] Negotiation successful
0810 23:32:49.503903 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3897,"thread_start_us":1830,"threads_started":1}
I20260810 23:32:49.506142 22190 tablet_service.cc:1467] Processing CreateTablet for tablet c8ad438cd73e4f9ea268b855802da0e1 (DEFAULT_TABLE table=test-table [id=b8a222ced20a4c0f92c8f1b69acbbc75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:49.506314 22403 tablet_service.cc:1467] Processing CreateTablet for tablet c8ad438cd73e4f9ea268b855802da0e1 (DEFAULT_TABLE table=test-table [id=b8a222ced20a4c0f92c8f1b69acbbc75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:49.507431 22190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ad438cd73e4f9ea268b855802da0e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.507722 22403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ad438cd73e4f9ea268b855802da0e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.507473 21968 tablet_service.cc:1467] Processing CreateTablet for tablet c8ad438cd73e4f9ea268b855802da0e1 (DEFAULT_TABLE table=test-table [id=b8a222ced20a4c0f92c8f1b69acbbc75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:49.508857 21968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8ad438cd73e4f9ea268b855802da0e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.519131 22559 tablet_bootstrap.cc:492] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb: Bootstrap starting.
I20260810 23:32:49.522400 22560 tablet_bootstrap.cc:492] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71: Bootstrap starting.
I20260810 23:32:49.522558 22559 tablet_bootstrap.cc:654] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:49.523988 22559 log.cc:824] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:49.527241 22560 tablet_bootstrap.cc:654] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:49.528820 22560 log.cc:824] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:49.529301 22561 tablet_bootstrap.cc:492] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb: Bootstrap starting.
I20260810 23:32:49.531847 22559 tablet_bootstrap.cc:492] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb: No bootstrap required, opened a new log
I20260810 23:32:49.532188 22559 ts_tablet_manager.cc:1397] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.012s
I20260810 23:32:49.532202 22561 tablet_bootstrap.cc:654] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:49.533578 22561 log.cc:824] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:49.534976 22560 tablet_bootstrap.cc:492] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71: No bootstrap required, opened a new log
I20260810 23:32:49.535244 22560 ts_tablet_manager.cc:1397] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260810 23:32:49.538657 22561 tablet_bootstrap.cc:492] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb: No bootstrap required, opened a new log
I20260810 23:32:49.538923 22561 ts_tablet_manager.cc:1397] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260810 23:32:49.542986 22559 raft_consensus.cc:348] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } }
I20260810 23:32:49.543404 22559 raft_consensus.cc:374] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.543509 22559 raft_consensus.cc:729] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8049a54497984825b0a1d9a4c5494ffb, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.544176 22559 consensus_queue.cc:260] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } }
I20260810 23:32:49.546254 22560 raft_consensus.cc:348] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } }
I20260810 23:32:49.546784 22560 raft_consensus.cc:374] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.546901 22560 raft_consensus.cc:729] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad8440266c9d4a34b3e431c523f79a71, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.547657 22561 raft_consensus.cc:348] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } }
I20260810 23:32:49.547623 22560 consensus_queue.cc:260] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } }
I20260810 23:32:49.548028 22561 raft_consensus.cc:374] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.548122 22561 raft_consensus.cc:729] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4418ca816004f7a8734905d079185fb, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.548980 22526 heartbeater.cc:503] Master 127.9.48.254:42567 was elected leader, sending a full tablet report...
I20260810 23:32:49.548866 22561 consensus_queue.cc:260] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } }
I20260810 23:32:49.550647 22561 ts_tablet_manager.cc:1428] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260810 23:32:49.549724 22560 ts_tablet_manager.cc:1428] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260810 23:32:49.552227 22295 heartbeater.cc:503] Master 127.9.48.254:42567 was elected leader, sending a full tablet report...
I20260810 23:32:49.552925 22559 ts_tablet_manager.cc:1428] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb: Time spent starting tablet: real 0.021s user 0.012s sys 0.007s
W20260810 23:32:49.563448 22296 tablet.cc:2369] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:32:49.570884 22567 raft_consensus.cc:482] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:49.571178 22567 raft_consensus.cc:504] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } }
I20260810 23:32:49.572782 22567 leader_election.cc:288] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8049a54497984825b0a1d9a4c5494ffb (127.9.48.194:44465), c4418ca816004f7a8734905d079185fb (127.9.48.193:40931)
I20260810 23:32:49.578774 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.573345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:49061 (local address 127.9.48.193:40931)
0810 23:32:49.573721 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:49.573725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:49.573739 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:32:49.573822 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:49.573829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:49.573930 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:49.574039 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:49.574511 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.575247 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.576936 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.577859 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.578015 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.578096 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:49.578416 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 23:32:49.578547 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260810 23:32:49.578792 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.573079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:40931 (local address 127.9.48.195:49061)
0810 23:32:49.573391 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:49.573417 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:49.573551 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:49.574094 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:49.574099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:49.574140 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:49.574350 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.574359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.575390 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.575397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.576766 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.576782 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.578020 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.578025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.578222 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.578258 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:32:49.578371 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 23:32:49.578507 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":20}
I20260810 23:32:49.580078 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.573535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:44465 (local address 127.9.48.195:34239)
0810 23:32:49.574272 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:49.574286 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:49.574447 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:49.574756 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:49.574760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:49.574784 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:49.575079 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.575088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.576104 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.576109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.577523 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.577535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.579301 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.579308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.579481 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.579512 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 23:32:49.579605 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 23:32:49.579723 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":1,"thread_start_us":407,"threads_started":1}
I20260810 23:32:49.580102 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.573185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:34239 (local address 127.9.48.194:44465)
0810 23:32:49.573546 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:49.573552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:49.574383 (+ 831us) server_negotiation.cc:407] Connection header received
0810 23:32:49.574579 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:49.574584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:49.574646 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:49.574739 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:49.575240 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.575964 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.577735 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.578892 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.579062 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.579138 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:49.579657 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 23:32:49.579831 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260810 23:32:49.579908 22007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ad438cd73e4f9ea268b855802da0e1" candidate_uuid: "ad8440266c9d4a34b3e431c523f79a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4418ca816004f7a8734905d079185fb" is_pre_election: true
I20260810 23:32:49.580497 22007 raft_consensus.cc:2454] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad8440266c9d4a34b3e431c523f79a71 in term 0.
I20260810 23:32:49.581197 22333 leader_election.cc:302] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad8440266c9d4a34b3e431c523f79a71, c4418ca816004f7a8734905d079185fb; no voters:
I20260810 23:32:49.581339 22219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ad438cd73e4f9ea268b855802da0e1" candidate_uuid: "ad8440266c9d4a34b3e431c523f79a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8049a54497984825b0a1d9a4c5494ffb" is_pre_election: true
I20260810 23:32:49.581593 22567 raft_consensus.cc:2781] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:49.581739 22567 raft_consensus.cc:482] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:49.581869 22567 raft_consensus.cc:3037] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:49.581854 22219 raft_consensus.cc:2454] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad8440266c9d4a34b3e431c523f79a71 in term 0.
I20260810 23:32:49.584254 22567 raft_consensus.cc:504] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } }
I20260810 23:32:49.584932 22567 leader_election.cc:288] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [CANDIDATE]: Term 1 election: Requested vote from peers 8049a54497984825b0a1d9a4c5494ffb (127.9.48.194:44465), c4418ca816004f7a8734905d079185fb (127.9.48.193:40931)
I20260810 23:32:49.585289 22219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ad438cd73e4f9ea268b855802da0e1" candidate_uuid: "ad8440266c9d4a34b3e431c523f79a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8049a54497984825b0a1d9a4c5494ffb"
I20260810 23:32:49.585440 22007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8ad438cd73e4f9ea268b855802da0e1" candidate_uuid: "ad8440266c9d4a34b3e431c523f79a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4418ca816004f7a8734905d079185fb"
I20260810 23:32:49.585494 22219 raft_consensus.cc:3037] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:49.585631 22007 raft_consensus.cc:3037] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:49.587874 22219 raft_consensus.cc:2454] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad8440266c9d4a34b3e431c523f79a71 in term 1.
I20260810 23:32:49.588342 22331 leader_election.cc:302] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8049a54497984825b0a1d9a4c5494ffb, ad8440266c9d4a34b3e431c523f79a71; no voters:
I20260810 23:32:49.588660 22567 raft_consensus.cc:2781] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:49.589058 22007 raft_consensus.cc:2454] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad8440266c9d4a34b3e431c523f79a71 in term 1.
I20260810 23:32:49.589231 22567 raft_consensus.cc:686] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [term 1 LEADER]: Becoming Leader. State: Replica: ad8440266c9d4a34b3e431c523f79a71, State: Running, Role: LEADER
I20260810 23:32:49.589726 22567 consensus_queue.cc:237] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } }
I20260810 23:32:49.594528 21764 catalog_manager.cc:5495] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad8440266c9d4a34b3e431c523f79a71 (127.9.48.195). New cstate: current_term: 1 leader_uuid: "ad8440266c9d4a34b3e431c523f79a71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } health_report { overall_health: UNKNOWN } } }
W20260810 23:32:49.624866 22527 tablet.cc:2369] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:32:49.630165 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.624265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46738 (local address 127.9.48.195:38101)
0810 23:32:49.624494 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:49.624503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:49.624609 (+ 106us) server_negotiation.cc:407] Connection header received
0810 23:32:49.624850 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:49.624858 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:49.624975 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:49.625139 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:32:49.625491 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.626521 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.627869 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.628288 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.628401 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.628787 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:32:49.629523 (+ 736us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:32:49.629677 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:49.629724 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 23:32:49.629797 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:32:49.639043 22007 raft_consensus.cc:1261] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Refusing update from remote peer ad8440266c9d4a34b3e431c523f79a71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:49.639178 22219 raft_consensus.cc:1261] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Refusing update from remote peer ad8440266c9d4a34b3e431c523f79a71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:49.640169 22574 consensus_queue.cc:1035] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:49.640561 22567 consensus_queue.cc:1035] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:49.667678 21764 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:49.667917 21764 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:49.671460 21764 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184: Sending DeleteTablet for 3 replicas of tablet c8ad438cd73e4f9ea268b855802da0e1
I20260810 23:32:49.672612 22190 tablet_service.cc:1514] Processing DeleteTablet for tablet c8ad438cd73e4f9ea268b855802da0e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:40942
I20260810 23:32:49.672637 22403 tablet_service.cc:1514] Processing DeleteTablet for tablet c8ad438cd73e4f9ea268b855802da0e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:46734
I20260810 23:32:49.672840 21968 tablet_service.cc:1514] Processing DeleteTablet for tablet c8ad438cd73e4f9ea268b855802da0e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:49 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:39932
I20260810 23:32:49.674515 21764 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:49.675704 21764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:49.682768 22589 tablet_replica.cc:330] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71: stopping tablet replica
I20260810 23:32:49.683346 22589 raft_consensus.cc:2229] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:49.685405 22589 raft_consensus.cc:2258] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:49.686264 22403 tablet_service.cc:1467] Processing CreateTablet for tablet 75d2f7ed22c243aeb27247f6c0977f4f (DEFAULT_TABLE table=test-table [id=50d71960b6574700b00e34a4cb121f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:49.686333 21966 tablet_service.cc:1467] Processing CreateTablet for tablet 75d2f7ed22c243aeb27247f6c0977f4f (DEFAULT_TABLE table=test-table [id=50d71960b6574700b00e34a4cb121f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:49.686789 22403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d2f7ed22c243aeb27247f6c0977f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.687002 21966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d2f7ed22c243aeb27247f6c0977f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.687716 22592 tablet_replica.cc:330] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb: stopping tablet replica
I20260810 23:32:49.688269 22592 raft_consensus.cc:2229] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:49.688905 22592 raft_consensus.cc:2258] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:49.690529 22189 tablet_service.cc:1467] Processing CreateTablet for tablet 75d2f7ed22c243aeb27247f6c0977f4f (DEFAULT_TABLE table=test-table [id=50d71960b6574700b00e34a4cb121f94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:49.691286 22189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d2f7ed22c243aeb27247f6c0977f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.694099 22560 tablet_bootstrap.cc:492] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71: Bootstrap starting.
I20260810 23:32:49.694427 22588 tablet_replica.cc:330] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb: stopping tablet replica
I20260810 23:32:49.694520 22589 ts_tablet_manager.cc:1905] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:49.694849 22588 raft_consensus.cc:2229] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:49.695365 22588 raft_consensus.cc:2258] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:49.695930 22560 tablet_bootstrap.cc:654] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:49.697516 22592 ts_tablet_manager.cc:1905] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:49.698267 22588 ts_tablet_manager.cc:1905] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:49.701499 22559 tablet_bootstrap.cc:492] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb: Bootstrap starting.
I20260810 23:32:49.703327 22559 tablet_bootstrap.cc:654] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:49.704627 22561 tablet_bootstrap.cc:492] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb: Bootstrap starting.
I20260810 23:32:49.704962 22592 ts_tablet_manager.cc:1918] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:49.705116 22592 log.cc:1196] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/c8ad438cd73e4f9ea268b855802da0e1
I20260810 23:32:49.705596 22588 ts_tablet_manager.cc:1918] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:49.705775 22588 log.cc:1196] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/c8ad438cd73e4f9ea268b855802da0e1
I20260810 23:32:49.705789 22592 ts_tablet_manager.cc:1939] T c8ad438cd73e4f9ea268b855802da0e1 P c4418ca816004f7a8734905d079185fb: Deleting consensus metadata
I20260810 23:32:49.706466 22588 ts_tablet_manager.cc:1939] T c8ad438cd73e4f9ea268b855802da0e1 P 8049a54497984825b0a1d9a4c5494ffb: Deleting consensus metadata
I20260810 23:32:49.706581 22561 tablet_bootstrap.cc:654] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:49.707954 21740 catalog_manager.cc:4841] TS c4418ca816004f7a8734905d079185fb (127.9.48.193:40931): tablet c8ad438cd73e4f9ea268b855802da0e1 (table test-table [id=b8a222ced20a4c0f92c8f1b69acbbc75]) successfully deleted
I20260810 23:32:49.708351 21738 catalog_manager.cc:4841] TS 8049a54497984825b0a1d9a4c5494ffb (127.9.48.194:44465): tablet c8ad438cd73e4f9ea268b855802da0e1 (table test-table [id=b8a222ced20a4c0f92c8f1b69acbbc75]) successfully deleted
I20260810 23:32:49.710641 22559 tablet_bootstrap.cc:492] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb: No bootstrap required, opened a new log
I20260810 23:32:49.710803 22559 ts_tablet_manager.cc:1397] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260810 23:32:49.710868 22561 tablet_bootstrap.cc:492] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb: No bootstrap required, opened a new log
I20260810 23:32:49.711005 22561 ts_tablet_manager.cc:1397] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:32:49.711722 22559 raft_consensus.cc:348] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.711829 22561 raft_consensus.cc:348] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.711978 22559 raft_consensus.cc:374] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.712065 22561 raft_consensus.cc:374] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.712075 22559 raft_consensus.cc:729] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8049a54497984825b0a1d9a4c5494ffb, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.712172 22561 raft_consensus.cc:729] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4418ca816004f7a8734905d079185fb, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.712440 22561 consensus_queue.cc:260] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.712440 22559 consensus_queue.cc:260] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.713106 22561 ts_tablet_manager.cc:1428] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:32:49.714133 22559 ts_tablet_manager.cc:1428] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:49.716291 22589 ts_tablet_manager.cc:1918] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:49.716465 22589 log.cc:1196] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/c8ad438cd73e4f9ea268b855802da0e1
I20260810 23:32:49.717120 22589 ts_tablet_manager.cc:1939] T c8ad438cd73e4f9ea268b855802da0e1 P ad8440266c9d4a34b3e431c523f79a71: Deleting consensus metadata
I20260810 23:32:49.718871 21739 catalog_manager.cc:4841] TS ad8440266c9d4a34b3e431c523f79a71 (127.9.48.195:38101): tablet c8ad438cd73e4f9ea268b855802da0e1 (table test-table [id=b8a222ced20a4c0f92c8f1b69acbbc75]) successfully deleted
I20260810 23:32:49.719053 22560 tablet_bootstrap.cc:492] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71: No bootstrap required, opened a new log
I20260810 23:32:49.719214 22560 ts_tablet_manager.cc:1397] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.001s
I20260810 23:32:49.720064 22560 raft_consensus.cc:348] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.720299 22560 raft_consensus.cc:374] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.720410 22560 raft_consensus.cc:729] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad8440266c9d4a34b3e431c523f79a71, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.720706 22560 consensus_queue.cc:260] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.721376 22560 ts_tablet_manager.cc:1428] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:32:49.731001 22567 raft_consensus.cc:482] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:49.731160 22567 raft_consensus.cc:504] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.731887 22567 leader_election.cc:288] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4418ca816004f7a8734905d079185fb (127.9.48.193:40931), 8049a54497984825b0a1d9a4c5494ffb (127.9.48.194:44465)
I20260810 23:32:49.732378 22007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d2f7ed22c243aeb27247f6c0977f4f" candidate_uuid: "ad8440266c9d4a34b3e431c523f79a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4418ca816004f7a8734905d079185fb" is_pre_election: true
I20260810 23:32:49.732463 22219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d2f7ed22c243aeb27247f6c0977f4f" candidate_uuid: "ad8440266c9d4a34b3e431c523f79a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8049a54497984825b0a1d9a4c5494ffb" is_pre_election: true
I20260810 23:32:49.732671 22007 raft_consensus.cc:2454] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad8440266c9d4a34b3e431c523f79a71 in term 0.
I20260810 23:32:49.732792 22219 raft_consensus.cc:2454] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad8440266c9d4a34b3e431c523f79a71 in term 0.
I20260810 23:32:49.733196 22331 leader_election.cc:302] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8049a54497984825b0a1d9a4c5494ffb, ad8440266c9d4a34b3e431c523f79a71; no voters:
I20260810 23:32:49.733489 22567 raft_consensus.cc:2781] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:49.733655 22567 raft_consensus.cc:482] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:49.733789 22567 raft_consensus.cc:3037] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:49.735996 22567 raft_consensus.cc:504] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.736652 22567 leader_election.cc:288] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [CANDIDATE]: Term 1 election: Requested vote from peers c4418ca816004f7a8734905d079185fb (127.9.48.193:40931), 8049a54497984825b0a1d9a4c5494ffb (127.9.48.194:44465)
I20260810 23:32:49.736984 22007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d2f7ed22c243aeb27247f6c0977f4f" candidate_uuid: "ad8440266c9d4a34b3e431c523f79a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4418ca816004f7a8734905d079185fb"
I20260810 23:32:49.737179 22007 raft_consensus.cc:3037] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:49.737154 22219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d2f7ed22c243aeb27247f6c0977f4f" candidate_uuid: "ad8440266c9d4a34b3e431c523f79a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8049a54497984825b0a1d9a4c5494ffb"
I20260810 23:32:49.737353 22219 raft_consensus.cc:3037] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:49.739766 22219 raft_consensus.cc:2454] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad8440266c9d4a34b3e431c523f79a71 in term 1.
I20260810 23:32:49.740127 22007 raft_consensus.cc:2454] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad8440266c9d4a34b3e431c523f79a71 in term 1.
I20260810 23:32:49.740157 22331 leader_election.cc:302] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8049a54497984825b0a1d9a4c5494ffb, ad8440266c9d4a34b3e431c523f79a71; no voters:
I20260810 23:32:49.740664 22567 raft_consensus.cc:2781] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:49.740844 22567 raft_consensus.cc:686] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [term 1 LEADER]: Becoming Leader. State: Replica: ad8440266c9d4a34b3e431c523f79a71, State: Running, Role: LEADER
I20260810 23:32:49.741207 22567 consensus_queue.cc:237] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.743999 21766 catalog_manager.cc:5495] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad8440266c9d4a34b3e431c523f79a71 (127.9.48.195). New cstate: current_term: 1 leader_uuid: "ad8440266c9d4a34b3e431c523f79a71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } health_report { overall_health: HEALTHY } } }
W20260810 23:32:49.766600 22079 tablet.cc:2369] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 23:32:49 dist-test-slave-04bb krb5kdc[21701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404767, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.254@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
I20260810 23:32:49.810871 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.795156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:42567 (local address 127.0.0.1:33156)
0810 23:32:49.795845 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:49.795867 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:49.796013 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:49.796401 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:49.796404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:49.797170 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:49.797463 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.797473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.798924 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.798928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.799906 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.799917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.800898 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.800926 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.801039 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.801754 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:49.801776 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:49.804222 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:49.807966 (+ 3744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:49.807977 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:49.807982 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:49.808394 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:49.808833 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:49.808837 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:49.808839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:49.809005 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:49.809662 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:49.809672 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:49.809893 (+ 221us) client_negotiation.cc:769] Sending connection context
0810 23:32:49.810101 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 23:32:49.810386 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":415,"threads_started":1}
I20260810 23:32:49.810976 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.795152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.9.48.254:42567)
0810 23:32:49.795386 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:49.795395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:49.795979 (+ 584us) server_negotiation.cc:407] Connection header received
0810 23:32:49.796137 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:49.796144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:49.796231 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:49.796362 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:49.797619 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.798783 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.800056 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.800728 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.800857 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.804421 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:49.804427 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:49.804431 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:49.804468 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:49.807813 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:49.808508 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:49.808514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:49.808517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:49.808693 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:49.809129 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:49.809135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:49.809138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:49.809517 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:49.809800 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:49.810204 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 23:32:49.810565 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":80}
I20260810 23:32:49.817138 21766 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:33156:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:32:49.818291 21766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:49.826011 22403 tablet_service.cc:1467] Processing CreateTablet for tablet 7d30f2e4247a45fa9b47bd01c3f6dbe7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ebee1e3557af4ed78969663be28718ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:32:49.825954 21966 tablet_service.cc:1467] Processing CreateTablet for tablet 7d30f2e4247a45fa9b47bd01c3f6dbe7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ebee1e3557af4ed78969663be28718ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:32:49.826625 22403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d30f2e4247a45fa9b47bd01c3f6dbe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.826876 21966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d30f2e4247a45fa9b47bd01c3f6dbe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.826682 22189 tablet_service.cc:1467] Processing CreateTablet for tablet 7d30f2e4247a45fa9b47bd01c3f6dbe7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ebee1e3557af4ed78969663be28718ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:32:49.827189 22189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d30f2e4247a45fa9b47bd01c3f6dbe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:49.834383 22561 tablet_bootstrap.cc:492] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb: Bootstrap starting.
I20260810 23:32:49.834399 22560 tablet_bootstrap.cc:492] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71: Bootstrap starting.
I20260810 23:32:49.835172 22559 tablet_bootstrap.cc:492] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb: Bootstrap starting.
I20260810 23:32:49.836249 22561 tablet_bootstrap.cc:654] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:49.836266 22560 tablet_bootstrap.cc:654] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:49.836979 22559 tablet_bootstrap.cc:654] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:49.848781 22561 tablet_bootstrap.cc:492] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb: No bootstrap required, opened a new log
I20260810 23:32:49.848974 22561 ts_tablet_manager.cc:1397] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260810 23:32:49.849812 22561 raft_consensus.cc:348] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.850162 22561 raft_consensus.cc:374] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.850250 22561 raft_consensus.cc:729] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4418ca816004f7a8734905d079185fb, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.850486 22559 tablet_bootstrap.cc:492] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb: No bootstrap required, opened a new log
I20260810 23:32:49.850713 22559 ts_tablet_manager.cc:1397] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.001s
I20260810 23:32:49.850569 22561 consensus_queue.cc:260] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.851079 22568 tablet_replica.cc:439] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } }
I20260810 23:32:49.851485 22568 tablet_replica.cc:442] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:32:49.851760 22559 raft_consensus.cc:348] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.852069 22559 raft_consensus.cc:374] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.852207 22559 raft_consensus.cc:729] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8049a54497984825b0a1d9a4c5494ffb, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.852339 22560 tablet_bootstrap.cc:492] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71: No bootstrap required, opened a new log
I20260810 23:32:49.852545 22560 ts_tablet_manager.cc:1397] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.004s
I20260810 23:32:49.853339 22560 raft_consensus.cc:348] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.852712 22559 consensus_queue.cc:260] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.853636 22560 raft_consensus.cc:374] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:49.853790 22560 raft_consensus.cc:729] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad8440266c9d4a34b3e431c523f79a71, State: Initialized, Role: FOLLOWER
I20260810 23:32:49.854072 22560 consensus_queue.cc:260] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.854048 22566 tablet_replica.cc:439] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } }
I20260810 23:32:49.854476 22566 tablet_replica.cc:442] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:32:49.854511 22567 tablet_replica.cc:439] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } }
I20260810 23:32:49.854826 22567 tablet_replica.cc:442] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:32:49.855300 22559 ts_tablet_manager.cc:1428] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:32:49.856429 22560 ts_tablet_manager.cc:1428] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260810 23:32:49.858376 22561 ts_tablet_manager.cc:1428] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260810 23:32:49.941161 22566 raft_consensus.cc:482] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:49.941461 22566 raft_consensus.cc:504] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.943168 22566 leader_election.cc:288] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4418ca816004f7a8734905d079185fb (127.9.48.193:40931), ad8440266c9d4a34b3e431c523f79a71 (127.9.48.195:38101)
I20260810 23:32:49.952570 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.943566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:53581 (local address 127.9.48.195:38101)
0810 23:32:49.943960 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:49.943967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:49.944597 (+ 630us) server_negotiation.cc:407] Connection header received
0810 23:32:49.944962 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:49.944990 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:49.945079 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:49.945175 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:49.946723 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.947928 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.950423 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.951333 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.951553 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.951650 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:49.952083 (+ 433us) server_negotiation.cc:299] Negotiation successful
0810 23:32:49.952229 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:32:49.952618 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.943518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:38101 (local address 127.9.48.194:53581)
0810 23:32:49.944453 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:49.944473 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:49.944740 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:49.945451 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:49.945455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:49.945483 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:49.946472 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.946496 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.948102 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.948107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.950252 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.950266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.951605 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.951611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.951811 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.951860 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 23:32:49.952021 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 23:32:49.952175 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":58,"thread_start_us":1011,"threads_started":1}
I20260810 23:32:49.952750 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.944045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:56215 (local address 127.9.48.193:40931)
0810 23:32:49.944252 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:49.944259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:49.945302 (+ 1043us) server_negotiation.cc:407] Connection header received
0810 23:32:49.945609 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:49.945617 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:49.945724 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:49.945801 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:32:49.946699 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.947478 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.950289 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.951644 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.951821 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.951911 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:49.952243 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 23:32:49.952396 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:32:49.953508 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:49.943547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:40931 (local address 127.9.48.194:56215)
0810 23:32:49.945209 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:49.945227 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:49.945428 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:49.946032 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:49.946036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:49.946062 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:32:49.946521 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.946531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.947631 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.947638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.950047 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:49.950064 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:49.951864 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:49.951870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:49.952062 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:49.952092 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:32:49.953106 (+ 1014us) client_negotiation.cc:240] Negotiation successful
0810 23:32:49.953238 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1499,"mutex_wait_us":128,"thread_start_us":2043,"threads_started":1}
I20260810 23:32:49.953574 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d30f2e4247a45fa9b47bd01c3f6dbe7" candidate_uuid: "8049a54497984825b0a1d9a4c5494ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8440266c9d4a34b3e431c523f79a71" is_pre_election: true
I20260810 23:32:49.954043 22444 raft_consensus.cc:2454] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8049a54497984825b0a1d9a4c5494ffb in term 0.
I20260810 23:32:49.954769 22111 leader_election.cc:302] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8049a54497984825b0a1d9a4c5494ffb, ad8440266c9d4a34b3e431c523f79a71; no voters:
I20260810 23:32:49.955052 22007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d30f2e4247a45fa9b47bd01c3f6dbe7" candidate_uuid: "8049a54497984825b0a1d9a4c5494ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4418ca816004f7a8734905d079185fb" is_pre_election: true
I20260810 23:32:49.955236 22566 raft_consensus.cc:2781] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:49.955372 22566 raft_consensus.cc:482] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:49.955358 22007 raft_consensus.cc:2454] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8049a54497984825b0a1d9a4c5494ffb in term 0.
I20260810 23:32:49.955478 22566 raft_consensus.cc:3037] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:49.958822 22566 raft_consensus.cc:504] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.959700 22566 leader_election.cc:288] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [CANDIDATE]: Term 1 election: Requested vote from peers c4418ca816004f7a8734905d079185fb (127.9.48.193:40931), ad8440266c9d4a34b3e431c523f79a71 (127.9.48.195:38101)
I20260810 23:32:49.959930 22007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d30f2e4247a45fa9b47bd01c3f6dbe7" candidate_uuid: "8049a54497984825b0a1d9a4c5494ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4418ca816004f7a8734905d079185fb"
I20260810 23:32:49.960068 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d30f2e4247a45fa9b47bd01c3f6dbe7" candidate_uuid: "8049a54497984825b0a1d9a4c5494ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8440266c9d4a34b3e431c523f79a71"
I20260810 23:32:49.960177 22007 raft_consensus.cc:3037] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:49.960297 22444 raft_consensus.cc:3037] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:49.962357 22007 raft_consensus.cc:2454] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8049a54497984825b0a1d9a4c5494ffb in term 1.
I20260810 23:32:49.962795 22112 leader_election.cc:302] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8049a54497984825b0a1d9a4c5494ffb, c4418ca816004f7a8734905d079185fb; no voters:
I20260810 23:32:49.963085 22566 raft_consensus.cc:2781] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:49.963601 22566 raft_consensus.cc:686] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [term 1 LEADER]: Becoming Leader. State: Replica: 8049a54497984825b0a1d9a4c5494ffb, State: Running, Role: LEADER
I20260810 23:32:49.963696 22444 raft_consensus.cc:2454] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8049a54497984825b0a1d9a4c5494ffb in term 1.
I20260810 23:32:49.964046 22566 consensus_queue.cc:237] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:49.966423 22616 tablet_replica.cc:439] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb: TxnStatusTablet state changed. Reason: New leader 8049a54497984825b0a1d9a4c5494ffb. Latest consensus state: current_term: 1 leader_uuid: "8049a54497984825b0a1d9a4c5494ffb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } }
I20260810 23:32:49.966902 22616 tablet_replica.cc:442] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:32:49.968004 22620 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:32:49.968240 22620 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:32:49.968433 21762 catalog_manager.cc:5495] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8049a54497984825b0a1d9a4c5494ffb (127.9.48.194). New cstate: current_term: 1 leader_uuid: "8049a54497984825b0a1d9a4c5494ffb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } health_report { overall_health: UNKNOWN } } }
I20260810 23:32:50.012853 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:50.007233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40950 (local address 127.9.48.194:44465)
0810 23:32:50.007452 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:50.007459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:50.007479 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:32:50.007788 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:50.007792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:50.007896 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:50.008055 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:32:50.008437 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.009283 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.010567 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.010987 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.011128 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:50.011601 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:32:50.012150 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:32:50.012331 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:50.012392 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 23:32:50.012475 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 23:32:50.012863 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:50.007013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:44465 (local address 127.0.0.1:40950)
0810 23:32:50.007392 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:50.007412 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:50.007558 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:50.008069 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:50.008074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:50.008086 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:32:50.008296 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:50.008305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.009415 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.009421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:50.010419 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:50.010431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.011153 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.011161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:50.011335 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:50.011392 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:32:50.012272 (+ 880us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:32:50.012294 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:32:50.012372 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 23:32:50.012555 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260810 23:32:50.017181 22007 raft_consensus.cc:1261] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Refusing update from remote peer 8049a54497984825b0a1d9a4c5494ffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:50.017477 22444 raft_consensus.cc:1261] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71 [term 1 FOLLOWER]: Refusing update from remote peer 8049a54497984825b0a1d9a4c5494ffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:50.018359 22616 consensus_queue.cc:1035] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:50.018750 22566 consensus_queue.cc:1035] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:50.031054 22567 tablet_replica.cc:439] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71: TxnStatusTablet state changed. Reason: New leader 8049a54497984825b0a1d9a4c5494ffb. Latest consensus state: current_term: 1 leader_uuid: "8049a54497984825b0a1d9a4c5494ffb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } }
I20260810 23:32:50.031445 22567 tablet_replica.cc:442] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:32:50.035642 22616 tablet_replica.cc:439] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8049a54497984825b0a1d9a4c5494ffb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } }
I20260810 23:32:50.035933 22616 tablet_replica.cc:442] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:32:50.036326 22579 mvcc.cc:205] Tried to move back new op lower bound from 7317113937982795776 to 7317113937777319936. Current Snapshot: MvccSnapshot[applied={T|T < 7317113937982795776}]
I20260810 23:32:50.038009 22567 tablet_replica.cc:439] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8049a54497984825b0a1d9a4c5494ffb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } }
I20260810 23:32:50.038311 22567 tablet_replica.cc:442] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P ad8440266c9d4a34b3e431c523f79a71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:32:50.043304 22566 tablet_replica.cc:439] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8049a54497984825b0a1d9a4c5494ffb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } }
I20260810 23:32:50.043644 22566 tablet_replica.cc:442] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P 8049a54497984825b0a1d9a4c5494ffb: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:32:50.043829 22568 tablet_replica.cc:439] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb: TxnStatusTablet state changed. Reason: New leader 8049a54497984825b0a1d9a4c5494ffb. Latest consensus state: current_term: 1 leader_uuid: "8049a54497984825b0a1d9a4c5494ffb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } }
I20260810 23:32:50.044155 22568 tablet_replica.cc:442] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:32:50.045308 22580 mvcc.cc:205] Tried to move back new op lower bound from 7317113937982795776 to 7317113937777319936. Current Snapshot: MvccSnapshot[applied={T|T < 7317113937982795776}]
I20260810 23:32:50.046963 22568 tablet_replica.cc:439] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8049a54497984825b0a1d9a4c5494ffb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } } }
I20260810 23:32:50.047188 22568 tablet_replica.cc:442] T 7d30f2e4247a45fa9b47bd01c3f6dbe7 P c4418ca816004f7a8734905d079185fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:32:50.066655 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:50.060043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:48041 (local address 127.9.48.194:44465)
0810 23:32:50.060249 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:50.060263 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:50.061006 (+ 743us) server_negotiation.cc:407] Connection header received
0810 23:32:50.061243 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:50.061248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:50.061342 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:50.061445 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:32:50.061991 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.062934 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.064432 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.065154 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.065289 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:50.065875 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:32:50.065989 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:32:50.066090 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:50.066228 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 23:32:50.066310 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:32:50.066781 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:50.060087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:44465 (local address 127.9.48.195:48041)
0810 23:32:50.060895 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:50.060919 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:50.061086 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:50.061463 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:50.061468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:50.061480 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:32:50.061813 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:50.061823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.063087 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.063093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:50.064273 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:50.064287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.065466 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.065471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:50.065614 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:50.065681 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:32:50.066093 (+ 412us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:32:50.066108 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 23:32:50.066204 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 23:32:50.066347 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":466,"threads_started":1}
I20260810 23:32:50.076686 22219 raft_consensus.cc:1261] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Refusing update from remote peer ad8440266c9d4a34b3e431c523f79a71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:50.076777 22005 raft_consensus.cc:1261] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Refusing update from remote peer ad8440266c9d4a34b3e431c523f79a71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:50.077358 22574 consensus_queue.cc:1035] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:50.077894 22567 consensus_queue.cc:1035] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:50.093878 22577 mvcc.cc:205] Tried to move back new op lower bound from 7317113938228985856 to 7317113936862437376. Current Snapshot: MvccSnapshot[applied={T|T < 7317113938228985856}]
I20260810 23:32:50.154871 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:50.147139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:42455 (local address 127.9.48.195:38101)
0810 23:32:50.147345 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:50.147352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:50.149193 (+ 1841us) server_negotiation.cc:407] Connection header received
0810 23:32:50.149422 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:50.149426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:50.149533 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:50.149639 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:32:50.150274 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.151099 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.152637 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.153146 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.153319 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:50.153900 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:32:50.154033 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:32:50.154112 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:50.154400 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 23:32:50.154490 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:32:50.156003 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:50.146057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:38101 (local address 127.9.48.194:42455)
0810 23:32:50.149046 (+ 2989us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:50.149079 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:50.149252 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:50.149668 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:50.149672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:50.149685 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:32:50.150024 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:50.150033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.151241 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.151247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:50.152445 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:50.152457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.153353 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.153362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:50.153571 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:50.153665 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:32:50.154203 (+ 538us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:32:50.154214 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 23:32:50.155481 (+ 1267us) client_negotiation.cc:240] Negotiation successful
0810 23:32:50.155598 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2754,"thread_start_us":2776,"threads_started":1}
I20260810 23:32:50.218034 21766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:50.218326 21766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:50.221820 21766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184: Sending DeleteTablet for 3 replicas of tablet 75d2f7ed22c243aeb27247f6c0977f4f
I20260810 23:32:50.222723 21966 tablet_service.cc:1514] Processing DeleteTablet for tablet 75d2f7ed22c243aeb27247f6c0977f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:50 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:39932
I20260810 23:32:50.223335 22189 tablet_service.cc:1514] Processing DeleteTablet for tablet 75d2f7ed22c243aeb27247f6c0977f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:50 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:40942
I20260810 23:32:50.223371 22403 tablet_service.cc:1514] Processing DeleteTablet for tablet 75d2f7ed22c243aeb27247f6c0977f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:50 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:46734
I20260810 23:32:50.248035 22647 tablet_replica.cc:330] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb: stopping tablet replica
I20260810 23:32:50.248585 22647 raft_consensus.cc:2229] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:50.248996 22647 raft_consensus.cc:2258] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:50.249032 22650 tablet_replica.cc:330] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb: stopping tablet replica
I20260810 23:32:50.249265 22650 raft_consensus.cc:2229] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:50.249552 22650 raft_consensus.cc:2258] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:50.250756 22647 ts_tablet_manager.cc:1905] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:50.252055 22650 ts_tablet_manager.cc:1905] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:50.256140 22649 tablet_replica.cc:330] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71: stopping tablet replica
I20260810 23:32:50.256358 22649 raft_consensus.cc:2229] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:50.256780 22649 raft_consensus.cc:2258] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:50.259332 22647 ts_tablet_manager.cc:1918] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:32:50.259490 22647 log.cc:1196] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/75d2f7ed22c243aeb27247f6c0977f4f
I20260810 23:32:50.260003 22647 ts_tablet_manager.cc:1939] T 75d2f7ed22c243aeb27247f6c0977f4f P c4418ca816004f7a8734905d079185fb: Deleting consensus metadata
I20260810 23:32:50.261351 21740 catalog_manager.cc:4841] TS c4418ca816004f7a8734905d079185fb (127.9.48.193:40931): tablet 75d2f7ed22c243aeb27247f6c0977f4f (table test-table [id=50d71960b6574700b00e34a4cb121f94]) successfully deleted
I20260810 23:32:50.262455 22650 ts_tablet_manager.cc:1918] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:32:50.262598 22650 log.cc:1196] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/75d2f7ed22c243aeb27247f6c0977f4f
I20260810 23:32:50.262909 22649 ts_tablet_manager.cc:1905] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:50.263096 22650 ts_tablet_manager.cc:1939] T 75d2f7ed22c243aeb27247f6c0977f4f P 8049a54497984825b0a1d9a4c5494ffb: Deleting consensus metadata
I20260810 23:32:50.264294 21738 catalog_manager.cc:4841] TS 8049a54497984825b0a1d9a4c5494ffb (127.9.48.194:44465): tablet 75d2f7ed22c243aeb27247f6c0977f4f (table test-table [id=50d71960b6574700b00e34a4cb121f94]) successfully deleted
I20260810 23:32:50.271443 22649 ts_tablet_manager.cc:1918] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:32:50.271652 22649 log.cc:1196] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/75d2f7ed22c243aeb27247f6c0977f4f
I20260810 23:32:50.272197 22649 ts_tablet_manager.cc:1939] T 75d2f7ed22c243aeb27247f6c0977f4f P ad8440266c9d4a34b3e431c523f79a71: Deleting consensus metadata
I20260810 23:32:50.273392 21739 catalog_manager.cc:4841] TS ad8440266c9d4a34b3e431c523f79a71 (127.9.48.195:38101): tablet 75d2f7ed22c243aeb27247f6c0977f4f (table test-table [id=50d71960b6574700b00e34a4cb121f94]) successfully deleted
Aug 10 23:32:50 dist-test-slave-04bb krb5kdc[21701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:32:50 dist-test-slave-04bb krb5kdc[21701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:32:50 dist-test-slave-04bb krb5kdc[21701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:32:50.328358 22660 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:50.317670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.9.48.254:42567)
0810 23:32:50.318486 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:50.318497 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:50.318512 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:32:50.318567 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:50.318571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:50.318644 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:50.318779 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:50.320113 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.320940 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.322196 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.322622 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.322776 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:50.328055 (+ 5279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33162: BlockingRecv error: recv got EOF from 127.0.0.1:33162 (error 108)
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":517,"threads_started":1}
Aug 10 23:32:50 dist-test-slave-04bb krb5kdc[21701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM
I20260810 23:32:50.348963 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:50.331668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.9.48.254:42567)
0810 23:32:50.331962 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:50.331968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:50.332284 (+ 316us) server_negotiation.cc:407] Connection header received
0810 23:32:50.332499 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:50.332505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:50.332622 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:50.332703 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:50.334326 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.335064 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.337000 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.337417 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.337516 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:50.342565 (+ 5049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:50.342572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:50.342578 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:50.342616 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:50.345812 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:50.346630 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:50.346637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:50.346641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:50.346806 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:50.347305 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:50.347309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:50.347311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:50.347633 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:50.347862 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:50.348274 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 23:32:50.348466 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:32:50.352854 21766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:50.353560 21766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:50.361361 21966 tablet_service.cc:1467] Processing CreateTablet for tablet f622ad59300f41faa6cf71bbd1c0d5b1 (DEFAULT_TABLE table=test-table [id=8e6691571499470ab08d191867436f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:50.361466 22189 tablet_service.cc:1467] Processing CreateTablet for tablet f622ad59300f41faa6cf71bbd1c0d5b1 (DEFAULT_TABLE table=test-table [id=8e6691571499470ab08d191867436f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:50.361881 21966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f622ad59300f41faa6cf71bbd1c0d5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:50.362010 22189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f622ad59300f41faa6cf71bbd1c0d5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:50.367719 22672 tablet_bootstrap.cc:492] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb: Bootstrap starting.
I20260810 23:32:50.368824 22674 tablet_bootstrap.cc:492] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb: Bootstrap starting.
I20260810 23:32:50.368988 22403 tablet_service.cc:1467] Processing CreateTablet for tablet f622ad59300f41faa6cf71bbd1c0d5b1 (DEFAULT_TABLE table=test-table [id=8e6691571499470ab08d191867436f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:50.369410 22403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f622ad59300f41faa6cf71bbd1c0d5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:50.370965 22672 tablet_bootstrap.cc:654] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:50.370970 22674 tablet_bootstrap.cc:654] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:50.386072 22674 tablet_bootstrap.cc:492] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb: No bootstrap required, opened a new log
I20260810 23:32:50.386309 22674 ts_tablet_manager.cc:1397] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20260810 23:32:50.386781 22672 tablet_bootstrap.cc:492] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb: No bootstrap required, opened a new log
I20260810 23:32:50.387037 22672 ts_tablet_manager.cc:1397] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.000s
I20260810 23:32:50.387452 22674 raft_consensus.cc:348] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.387892 22674 raft_consensus.cc:374] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:50.388129 22674 raft_consensus.cc:729] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4418ca816004f7a8734905d079185fb, State: Initialized, Role: FOLLOWER
I20260810 23:32:50.388350 22672 raft_consensus.cc:348] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.388736 22672 raft_consensus.cc:374] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:50.388504 22674 consensus_queue.cc:260] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.389057 22672 raft_consensus.cc:729] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8049a54497984825b0a1d9a4c5494ffb, State: Initialized, Role: FOLLOWER
I20260810 23:32:50.389437 22672 consensus_queue.cc:260] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.390986 22672 ts_tablet_manager.cc:1428] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:32:50.392961 22674 ts_tablet_manager.cc:1428] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260810 23:32:50.393055 22677 tablet_bootstrap.cc:492] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71: Bootstrap starting.
I20260810 23:32:50.395061 22677 tablet_bootstrap.cc:654] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:50.397526 22677 tablet_bootstrap.cc:492] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71: No bootstrap required, opened a new log
I20260810 23:32:50.397689 22677 ts_tablet_manager.cc:1397] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:32:50.398573 22677 raft_consensus.cc:348] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.398844 22677 raft_consensus.cc:374] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:50.398947 22677 raft_consensus.cc:729] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad8440266c9d4a34b3e431c523f79a71, State: Initialized, Role: FOLLOWER
I20260810 23:32:50.399259 22677 consensus_queue.cc:260] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.401449 22677 ts_tablet_manager.cc:1428] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:50.526291 22574 raft_consensus.cc:482] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:50.526487 22574 raft_consensus.cc:504] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.527254 22574 leader_election.cc:288] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4418ca816004f7a8734905d079185fb (127.9.48.193:40931), 8049a54497984825b0a1d9a4c5494ffb (127.9.48.194:44465)
I20260810 23:32:50.527938 22005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f622ad59300f41faa6cf71bbd1c0d5b1" candidate_uuid: "ad8440266c9d4a34b3e431c523f79a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4418ca816004f7a8734905d079185fb" is_pre_election: true
I20260810 23:32:50.527971 22218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f622ad59300f41faa6cf71bbd1c0d5b1" candidate_uuid: "ad8440266c9d4a34b3e431c523f79a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8049a54497984825b0a1d9a4c5494ffb" is_pre_election: true
I20260810 23:32:50.528264 22005 raft_consensus.cc:2454] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad8440266c9d4a34b3e431c523f79a71 in term 0.
I20260810 23:32:50.528262 22218 raft_consensus.cc:2454] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad8440266c9d4a34b3e431c523f79a71 in term 0.
I20260810 23:32:50.528739 22333 leader_election.cc:302] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad8440266c9d4a34b3e431c523f79a71, c4418ca816004f7a8734905d079185fb; no voters:
I20260810 23:32:50.529074 22574 raft_consensus.cc:2781] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:50.529217 22574 raft_consensus.cc:482] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:50.529311 22574 raft_consensus.cc:3037] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:50.531553 22574 raft_consensus.cc:504] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.532205 22574 leader_election.cc:288] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [CANDIDATE]: Term 1 election: Requested vote from peers c4418ca816004f7a8734905d079185fb (127.9.48.193:40931), 8049a54497984825b0a1d9a4c5494ffb (127.9.48.194:44465)
I20260810 23:32:50.532758 22005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f622ad59300f41faa6cf71bbd1c0d5b1" candidate_uuid: "ad8440266c9d4a34b3e431c523f79a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4418ca816004f7a8734905d079185fb"
I20260810 23:32:50.532759 22218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f622ad59300f41faa6cf71bbd1c0d5b1" candidate_uuid: "ad8440266c9d4a34b3e431c523f79a71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8049a54497984825b0a1d9a4c5494ffb"
I20260810 23:32:50.533036 22005 raft_consensus.cc:3037] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:50.533026 22218 raft_consensus.cc:3037] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:50.535118 22005 raft_consensus.cc:2454] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad8440266c9d4a34b3e431c523f79a71 in term 1.
I20260810 23:32:50.535130 22218 raft_consensus.cc:2454] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad8440266c9d4a34b3e431c523f79a71 in term 1.
I20260810 23:32:50.535629 22333 leader_election.cc:302] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8049a54497984825b0a1d9a4c5494ffb, ad8440266c9d4a34b3e431c523f79a71, c4418ca816004f7a8734905d079185fb; no voters:
I20260810 23:32:50.536005 22574 raft_consensus.cc:2781] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:50.536170 22574 raft_consensus.cc:686] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [term 1 LEADER]: Becoming Leader. State: Replica: ad8440266c9d4a34b3e431c523f79a71, State: Running, Role: LEADER
I20260810 23:32:50.536468 22574 consensus_queue.cc:237] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.539302 21762 catalog_manager.cc:5495] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad8440266c9d4a34b3e431c523f79a71 (127.9.48.195). New cstate: current_term: 1 leader_uuid: "ad8440266c9d4a34b3e431c523f79a71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } health_report { overall_health: HEALTHY } } }
I20260810 23:32:50.571419 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:50.566565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.9.48.195:38101)
0810 23:32:50.566806 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:50.566812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:50.566855 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:32:50.567140 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:50.567145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:50.567219 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:50.567362 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:32:50.567723 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.568473 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.569760 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.570179 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.570274 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:50.570684 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:32:50.570794 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:32:50.570862 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:50.570999 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 23:32:50.571066 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 23:32:50.574219 22005 raft_consensus.cc:1261] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Refusing update from remote peer ad8440266c9d4a34b3e431c523f79a71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:50.574229 22218 raft_consensus.cc:1261] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Refusing update from remote peer ad8440266c9d4a34b3e431c523f79a71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:50.574914 22567 consensus_queue.cc:1035] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:50.575209 22574 consensus_queue.cc:1035] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:50.587673 21760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:50.587905 21760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:50.591123 21760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184: Sending DeleteTablet for 3 replicas of tablet f622ad59300f41faa6cf71bbd1c0d5b1
I20260810 23:32:50.592198 22189 tablet_service.cc:1514] Processing DeleteTablet for tablet f622ad59300f41faa6cf71bbd1c0d5b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:50 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:40942
I20260810 23:32:50.592474 22650 tablet_replica.cc:330] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb: stopping tablet replica
I20260810 23:32:50.593014 22650 raft_consensus.cc:2229] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:50.593235 22650 raft_consensus.cc:2258] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:50.594048 21966 tablet_service.cc:1514] Processing DeleteTablet for tablet f622ad59300f41faa6cf71bbd1c0d5b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:50 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:39932
I20260810 23:32:50.594337 22647 tablet_replica.cc:330] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb: stopping tablet replica
I20260810 23:32:50.594513 22647 raft_consensus.cc:2229] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:50.594699 22647 raft_consensus.cc:2258] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:50.594735 22403 tablet_service.cc:1514] Processing DeleteTablet for tablet f622ad59300f41faa6cf71bbd1c0d5b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:50 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:46734
I20260810 23:32:50.595029 22649 tablet_replica.cc:330] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71: stopping tablet replica
I20260810 23:32:50.595242 22649 raft_consensus.cc:2229] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:50.595542 22649 raft_consensus.cc:2258] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:50.595499 21762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:32:50.596146 21762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:32:50.597322 22649 ts_tablet_manager.cc:1905] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:50.601575 22647 ts_tablet_manager.cc:1905] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:50.603340 22403 tablet_service.cc:1467] Processing CreateTablet for tablet c6793ed08eb04a2cb7c14ed3f87743e5 (DEFAULT_TABLE table=test-table [id=a86bd3ff3a324b129f2b17a73ea4ea13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:50.603734 22403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6793ed08eb04a2cb7c14ed3f87743e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:50.604859 21966 tablet_service.cc:1467] Processing CreateTablet for tablet c6793ed08eb04a2cb7c14ed3f87743e5 (DEFAULT_TABLE table=test-table [id=a86bd3ff3a324b129f2b17a73ea4ea13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:50.605219 21966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6793ed08eb04a2cb7c14ed3f87743e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:50.608742 22677 tablet_bootstrap.cc:492] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71: Bootstrap starting.
I20260810 23:32:50.609181 22189 tablet_service.cc:1467] Processing CreateTablet for tablet c6793ed08eb04a2cb7c14ed3f87743e5 (DEFAULT_TABLE table=test-table [id=a86bd3ff3a324b129f2b17a73ea4ea13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:32:50.609615 22650 ts_tablet_manager.cc:1905] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:50.609611 22189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6793ed08eb04a2cb7c14ed3f87743e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:50.610404 22677 tablet_bootstrap.cc:654] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:50.613427 22647 ts_tablet_manager.cc:1918] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:50.613585 22647 log.cc:1196] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/f622ad59300f41faa6cf71bbd1c0d5b1
I20260810 23:32:50.614051 22647 ts_tablet_manager.cc:1939] T f622ad59300f41faa6cf71bbd1c0d5b1 P c4418ca816004f7a8734905d079185fb: Deleting consensus metadata
I20260810 23:32:50.615151 21740 catalog_manager.cc:4841] TS c4418ca816004f7a8734905d079185fb (127.9.48.193:40931): tablet f622ad59300f41faa6cf71bbd1c0d5b1 (table test-table [id=8e6691571499470ab08d191867436f14]) successfully deleted
I20260810 23:32:50.617718 22650 ts_tablet_manager.cc:1918] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:50.617903 22650 log.cc:1196] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/f622ad59300f41faa6cf71bbd1c0d5b1
I20260810 23:32:50.618361 22650 ts_tablet_manager.cc:1939] T f622ad59300f41faa6cf71bbd1c0d5b1 P 8049a54497984825b0a1d9a4c5494ffb: Deleting consensus metadata
I20260810 23:32:50.618440 22672 tablet_bootstrap.cc:492] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb: Bootstrap starting.
I20260810 23:32:50.619520 21738 catalog_manager.cc:4841] TS 8049a54497984825b0a1d9a4c5494ffb (127.9.48.194:44465): tablet f622ad59300f41faa6cf71bbd1c0d5b1 (table test-table [id=8e6691571499470ab08d191867436f14]) successfully deleted
I20260810 23:32:50.620119 22672 tablet_bootstrap.cc:654] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:50.621846 22677 tablet_bootstrap.cc:492] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71: No bootstrap required, opened a new log
I20260810 23:32:50.622054 22677 ts_tablet_manager.cc:1397] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260810 23:32:50.622706 22677 raft_consensus.cc:348] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.622954 22677 raft_consensus.cc:374] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:50.623049 22677 raft_consensus.cc:729] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad8440266c9d4a34b3e431c523f79a71, State: Initialized, Role: FOLLOWER
I20260810 23:32:50.623168 22649 ts_tablet_manager.cc:1918] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:32:50.623397 22649 log.cc:1196] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/f622ad59300f41faa6cf71bbd1c0d5b1
I20260810 23:32:50.623339 22677 consensus_queue.cc:260] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.623651 22674 tablet_bootstrap.cc:492] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb: Bootstrap starting.
I20260810 23:32:50.623865 22649 ts_tablet_manager.cc:1939] T f622ad59300f41faa6cf71bbd1c0d5b1 P ad8440266c9d4a34b3e431c523f79a71: Deleting consensus metadata
I20260810 23:32:50.625224 21739 catalog_manager.cc:4841] TS ad8440266c9d4a34b3e431c523f79a71 (127.9.48.195:38101): tablet f622ad59300f41faa6cf71bbd1c0d5b1 (table test-table [id=8e6691571499470ab08d191867436f14]) successfully deleted
I20260810 23:32:50.625458 22674 tablet_bootstrap.cc:654] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb: Neither blocks nor log segments found. Creating new log.
I20260810 23:32:50.626349 22677 ts_tablet_manager.cc:1428] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260810 23:32:50.627499 22674 tablet_bootstrap.cc:492] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb: No bootstrap required, opened a new log
I20260810 23:32:50.627682 22674 ts_tablet_manager.cc:1397] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 23:32:50.628640 22674 raft_consensus.cc:348] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.628953 22674 raft_consensus.cc:374] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:50.629047 22674 raft_consensus.cc:729] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4418ca816004f7a8734905d079185fb, State: Initialized, Role: FOLLOWER
I20260810 23:32:50.629401 22672 tablet_bootstrap.cc:492] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb: No bootstrap required, opened a new log
I20260810 23:32:50.629316 22674 consensus_queue.cc:260] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.629577 22672 ts_tablet_manager.cc:1397] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260810 23:32:50.629877 22674 ts_tablet_manager.cc:1428] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:32:50.630353 22672 raft_consensus.cc:348] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.630647 22672 raft_consensus.cc:374] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:50.630735 22672 raft_consensus.cc:729] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8049a54497984825b0a1d9a4c5494ffb, State: Initialized, Role: FOLLOWER
I20260810 23:32:50.631014 22672 consensus_queue.cc:260] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.631605 22672 ts_tablet_manager.cc:1428] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:32:50.763563 22616 raft_consensus.cc:482] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:32:50.763792 22616 raft_consensus.cc:504] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.764467 22616 leader_election.cc:288] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4418ca816004f7a8734905d079185fb (127.9.48.193:40931), ad8440266c9d4a34b3e431c523f79a71 (127.9.48.195:38101)
I20260810 23:32:50.765033 22005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6793ed08eb04a2cb7c14ed3f87743e5" candidate_uuid: "8049a54497984825b0a1d9a4c5494ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4418ca816004f7a8734905d079185fb" is_pre_election: true
I20260810 23:32:50.765084 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6793ed08eb04a2cb7c14ed3f87743e5" candidate_uuid: "8049a54497984825b0a1d9a4c5494ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8440266c9d4a34b3e431c523f79a71" is_pre_election: true
I20260810 23:32:50.765384 22444 raft_consensus.cc:2454] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8049a54497984825b0a1d9a4c5494ffb in term 0.
I20260810 23:32:50.765389 22005 raft_consensus.cc:2454] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8049a54497984825b0a1d9a4c5494ffb in term 0.
I20260810 23:32:50.765852 22111 leader_election.cc:302] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8049a54497984825b0a1d9a4c5494ffb, ad8440266c9d4a34b3e431c523f79a71; no voters:
I20260810 23:32:50.766139 22616 raft_consensus.cc:2781] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:32:50.766225 22616 raft_consensus.cc:482] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:32:50.766366 22616 raft_consensus.cc:3037] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:50.768332 22616 raft_consensus.cc:504] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.768988 22616 leader_election.cc:288] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [CANDIDATE]: Term 1 election: Requested vote from peers c4418ca816004f7a8734905d079185fb (127.9.48.193:40931), ad8440266c9d4a34b3e431c523f79a71 (127.9.48.195:38101)
I20260810 23:32:50.769363 22005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6793ed08eb04a2cb7c14ed3f87743e5" candidate_uuid: "8049a54497984825b0a1d9a4c5494ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4418ca816004f7a8734905d079185fb"
I20260810 23:32:50.769603 22005 raft_consensus.cc:3037] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:50.769528 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6793ed08eb04a2cb7c14ed3f87743e5" candidate_uuid: "8049a54497984825b0a1d9a4c5494ffb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8440266c9d4a34b3e431c523f79a71"
I20260810 23:32:50.769742 22444 raft_consensus.cc:3037] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:50.772696 22005 raft_consensus.cc:2454] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8049a54497984825b0a1d9a4c5494ffb in term 1.
I20260810 23:32:50.772713 22444 raft_consensus.cc:2454] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8049a54497984825b0a1d9a4c5494ffb in term 1.
I20260810 23:32:50.773119 22112 leader_election.cc:302] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8049a54497984825b0a1d9a4c5494ffb, c4418ca816004f7a8734905d079185fb; no voters:
I20260810 23:32:50.773406 22616 raft_consensus.cc:2781] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:50.773767 22616 raft_consensus.cc:686] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [term 1 LEADER]: Becoming Leader. State: Replica: 8049a54497984825b0a1d9a4c5494ffb, State: Running, Role: LEADER
I20260810 23:32:50.774039 22616 consensus_queue.cc:237] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } }
I20260810 23:32:50.776639 21762 catalog_manager.cc:5495] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8049a54497984825b0a1d9a4c5494ffb (127.9.48.194). New cstate: current_term: 1 leader_uuid: "8049a54497984825b0a1d9a4c5494ffb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8049a54497984825b0a1d9a4c5494ffb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 } health_report { overall_health: UNKNOWN } } }
I20260810 23:32:50.815655 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:50.809313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.9.48.194:44465)
0810 23:32:50.810018 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:50.810023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:50.810040 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:32:50.810095 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:50.810129 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:50.810198 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:50.810291 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:32:50.810750 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.811608 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.813542 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.814001 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.814134 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:50.814716 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:32:50.814840 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:32:50.814926 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:50.815164 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 23:32:50.815234 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":433,"threads_started":1}
I20260810 23:32:50.831341 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:50.824892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:34701 (local address 127.9.48.194:44465)
0810 23:32:50.825067 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:50.825073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:50.825669 (+ 596us) server_negotiation.cc:407] Connection header received
0810 23:32:50.825852 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:50.825856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:50.825943 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:50.826061 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:32:50.826561 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.827425 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.829429 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.829957 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.830066 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:50.830532 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:32:50.830640 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:32:50.830719 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:50.830902 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 23:32:50.830992 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":55}
I20260810 23:32:50.831749 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:50.824892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:44465 (local address 127.9.48.194:34701)
0810 23:32:50.825560 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:50.825578 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:50.825721 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:50.826076 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:50.826081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:50.826093 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:32:50.826399 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:50.826410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.827554 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.827559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:50.829263 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:50.829278 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.830134 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.830141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:50.830325 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:50.830370 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:32:50.830744 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:32:50.830755 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 23:32:50.830836 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:32:50.830988 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":1,"thread_start_us":376,"threads_started":1}
I20260810 23:32:50.840008 22444 raft_consensus.cc:1261] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71 [term 1 FOLLOWER]: Refusing update from remote peer 8049a54497984825b0a1d9a4c5494ffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:50.840065 22005 raft_consensus.cc:1261] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Refusing update from remote peer 8049a54497984825b0a1d9a4c5494ffb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:32:50.840685 22696 consensus_queue.cc:1035] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad8440266c9d4a34b3e431c523f79a71" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:50.841095 22616 consensus_queue.cc:1035] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4418ca816004f7a8734905d079185fb" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:32:50.903527 21760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:50.903851 21760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:32:50.907074 21760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e72b0b8f35d4432ba3391ead7780184: Sending DeleteTablet for 3 replicas of tablet c6793ed08eb04a2cb7c14ed3f87743e5
I20260810 23:32:50.907902 22189 tablet_service.cc:1514] Processing DeleteTablet for tablet c6793ed08eb04a2cb7c14ed3f87743e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:50 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:40942
I20260810 23:32:50.908025 21966 tablet_service.cc:1514] Processing DeleteTablet for tablet c6793ed08eb04a2cb7c14ed3f87743e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:50 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:39932
I20260810 23:32:50.908232 22650 tablet_replica.cc:330] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb: stopping tablet replica
I20260810 23:32:50.908322 22647 tablet_replica.cc:330] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb: stopping tablet replica
I20260810 23:32:50.908430 22650 raft_consensus.cc:2229] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:32:50.908504 22647 raft_consensus.cc:2229] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:50.908754 22647 raft_consensus.cc:2258] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:50.908717 22403 tablet_service.cc:1514] Processing DeleteTablet for tablet c6793ed08eb04a2cb7c14ed3f87743e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:32:50 UTC) from {username='oryx', principal='oryx/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:46734
I20260810 23:32:50.909001 22650 raft_consensus.cc:2258] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:50.909065 22649 tablet_replica.cc:330] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71: stopping tablet replica
I20260810 23:32:50.909250 22649 raft_consensus.cc:2229] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:32:50.909487 22649 raft_consensus.cc:2258] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:32:50.910791 22647 ts_tablet_manager.cc:1905] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:50.910912 22650 ts_tablet_manager.cc:1905] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:50.911221 22649 ts_tablet_manager.cc:1905] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:32:50.911476 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 21877
I20260810 23:32:50.918056 22650 ts_tablet_manager.cc:1918] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:32:50.918231 22650 log.cc:1196] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/c6793ed08eb04a2cb7c14ed3f87743e5
I20260810 23:32:50.918589 22649 ts_tablet_manager.cc:1918] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:32:50.918741 22650 ts_tablet_manager.cc:1939] T c6793ed08eb04a2cb7c14ed3f87743e5 P 8049a54497984825b0a1d9a4c5494ffb: Deleting consensus metadata
I20260810 23:32:50.918754 22649 log.cc:1196] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/c6793ed08eb04a2cb7c14ed3f87743e5
I20260810 23:32:50.918982 22647 ts_tablet_manager.cc:1918] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:32:50.919174 22647 log.cc:1196] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/c6793ed08eb04a2cb7c14ed3f87743e5
I20260810 23:32:50.919294 22649 ts_tablet_manager.cc:1939] T c6793ed08eb04a2cb7c14ed3f87743e5 P ad8440266c9d4a34b3e431c523f79a71: Deleting consensus metadata
I20260810 23:32:50.919806 22647 ts_tablet_manager.cc:1939] T c6793ed08eb04a2cb7c14ed3f87743e5 P c4418ca816004f7a8734905d079185fb: Deleting consensus metadata
I20260810 23:32:50.920084 21738 catalog_manager.cc:4841] TS 8049a54497984825b0a1d9a4c5494ffb (127.9.48.194:44465): tablet c6793ed08eb04a2cb7c14ed3f87743e5 (table test-table [id=a86bd3ff3a324b129f2b17a73ea4ea13]) successfully deleted
I20260810 23:32:50.920603 21739 catalog_manager.cc:4841] TS ad8440266c9d4a34b3e431c523f79a71 (127.9.48.195:38101): tablet c6793ed08eb04a2cb7c14ed3f87743e5 (table test-table [id=a86bd3ff3a324b129f2b17a73ea4ea13]) successfully deleted
I20260810 23:32:50.921423 21740 catalog_manager.cc:4841] TS c4418ca816004f7a8734905d079185fb (127.9.48.193:40931): tablet c6793ed08eb04a2cb7c14ed3f87743e5 (table test-table [id=a86bd3ff3a324b129f2b17a73ea4ea13]) successfully deleted
Aug 10 23:32:50 dist-test-slave-04bb krb5kdc[21701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.48.193@KRBTEST.COM
I20260810 23:32:50.937580 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:50.917872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.9.48.193:40931)
0810 23:32:50.921393 (+ 3521us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:50.921399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:50.921420 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:32:50.921485 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:50.921491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:50.921619 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:50.921740 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:50.922981 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.924259 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.925569 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:50.926303 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:50.926461 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:50.930797 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:50.930819 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:50.930842 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:50.930888 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:50.934474 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:50.935190 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:50.935197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:50.935200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:50.935361 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:50.935772 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:50.935776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:50.935778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:50.936133 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:50.936475 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:50.936996 (+ 521us) server_negotiation.cc:299] Negotiation successful
0810 23:32:50.937226 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3390,"thread_start_us":3272,"threads_started":1}
I20260810 23:32:50.938234 22068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:51.066597 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 21877
I20260810 23:32:51.091629 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 22083
Aug 10 23:32:51 dist-test-slave-04bb krb5kdc[21701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.48.194@KRBTEST.COM
I20260810 23:32:51.110037 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:51.092568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40964 (local address 127.9.48.194:44465)
0810 23:32:51.092818 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:51.092825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:51.092849 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:32:51.092925 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:51.092931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:51.093019 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:51.093171 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:51.094433 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:51.095772 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:51.097235 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:51.097999 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:51.098147 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:51.102315 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:51.102336 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:51.102358 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:51.102402 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:51.106845 (+ 4443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:51.107606 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:51.107619 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:51.107623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:51.107816 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:51.108212 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:51.108218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:51.108223 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:51.108771 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:51.109161 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:51.109350 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 23:32:51.109606 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 23:32:51.110965 22290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:51.260764 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 22083
I20260810 23:32:51.294942 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 22300
Aug 10 23:32:51 dist-test-slave-04bb krb5kdc[21701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.48.195@KRBTEST.COM
I20260810 23:32:51.310022 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:51.295672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.9.48.195:38101)
0810 23:32:51.296154 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:51.296159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:51.296176 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:32:51.296235 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:51.296239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:51.296305 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:51.296398 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:51.297700 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:51.298394 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:51.299621 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:51.300028 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:51.300123 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:51.304593 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:51.304610 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:51.304627 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:51.304666 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:51.307588 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:51.308195 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:51.308203 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:51.308205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:51.308364 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:51.308791 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:51.308794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:51.308796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:51.309150 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:51.309390 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:51.309600 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 23:32:51.309760 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":239,"threads_started":1}
I20260810 23:32:51.310703 22518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:51.436870 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 22300
I20260810 23:32:51.473256 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 21719
I20260810 23:32:51.474506 21826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:51.595321 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 21719
2026-08-10T23:32:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6057 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:32:51 dist-test-slave-04bb krb5kdc[22725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:32:51 dist-test-slave-04bb krb5kdc[22725](info): set up 2 sockets
Aug 10 23:32:51 dist-test-slave-04bb krb5kdc[22725](info): commencing operation
krb5kdc: starting...
W20260810 23:32:52.952644 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.301s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:32:53 dist-test-slave-04bb krb5kdc[22725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:32:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:32:53.224623 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:46729
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:40423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:46729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:53.556777 22746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:53.557112 22746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:53.557288 22746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:53.564355 22746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:53.564492 22746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:53.564628 22746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:53.564733 22746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:53.564828 22746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:53.574460 22746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:40423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:46729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:46729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:53.575804 22746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:53.577170 22746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:53.589172 22754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:53.589210 22755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:53.589265 22757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:53.590365 22746 server_base.cc:1034] running on GCE node
I20260810 23:32:53.591142 22746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:53.592589 22746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:53.593848 22746 hybrid_clock.cc:648] HybridClock initialized: now 1786404773593795 us; error 58 us; skew 500 ppm
I20260810 23:32:53.594213 22746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T23:32:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2060 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:32:53 dist-test-slave-04bb krb5kdc[22769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:32:53 dist-test-slave-04bb krb5kdc[22769](info): set up 2 sockets
Aug 10 23:32:53 dist-test-slave-04bb krb5kdc[22769](info): commencing operation
krb5kdc: starting...
W20260810 23:32:54.992156 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.280s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:32:55 dist-test-slave-04bb krb5kdc[22769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:32:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:32:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:32:55.264038 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:39771
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:39771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:55.602772 22798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:55.603050 22798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:55.603210 22798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:55.610311 22798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:55.610412 22798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:55.610462 22798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:55.610543 22798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:55.610664 22798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:55.620893 22798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:39771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:39771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:55.622239 22798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:55.623955 22798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:55.635743 22817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:55.635805 22812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:55.635782 22813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:55.636845 22798 server_base.cc:1034] running on GCE node
I20260810 23:32:55.637552 22798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:55.638960 22798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:55.640164 22798 hybrid_clock.cc:648] HybridClock initialized: now 1786404775640093 us; error 61 us; skew 500 ppm
I20260810 23:32:55.640514 22798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:55 dist-test-slave-04bb krb5kdc[22769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404775, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:55.643687 22798 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:32:55.645191 22798 webserver.cc:460] Webserver started at http://127.9.48.254:42119/ using document root <none> and password file <none>
I20260810 23:32:55.645852 22798 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:55.645951 22798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:55.646211 22798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:55.649315 22798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "45363f6dee624488ae24ccbfb4478e98"
format_stamp: "Formatted at 2026-08-10 23:32:55 on dist-test-slave-04bb"
server_key: "c4f44a56198f5145d3fdb4b2f723f1be"
server_key_iv: "796a000435b37d56d886dbc8e830fc82"
server_key_version: "encryptionkey@0"
I20260810 23:32:55.650028 22798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "45363f6dee624488ae24ccbfb4478e98"
format_stamp: "Formatted at 2026-08-10 23:32:55 on dist-test-slave-04bb"
server_key: "c4f44a56198f5145d3fdb4b2f723f1be"
server_key_iv: "796a000435b37d56d886dbc8e830fc82"
server_key_version: "encryptionkey@0"
I20260810 23:32:55.655000 22798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:32:55.658556 22826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:55.659942 22798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:32:55.660111 22798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "45363f6dee624488ae24ccbfb4478e98"
format_stamp: "Formatted at 2026-08-10 23:32:55 on dist-test-slave-04bb"
server_key: "c4f44a56198f5145d3fdb4b2f723f1be"
server_key_iv: "796a000435b37d56d886dbc8e830fc82"
server_key_version: "encryptionkey@0"
I20260810 23:32:55.660287 22798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:55.678299 22798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:55.682411 22798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:55.682703 22798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:55.701906 22798 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:39771
I20260810 23:32:55.702003 22904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:39771 every 8 connection(s)
I20260810 23:32:55.703392 22798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:55.707418 22905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:32:55.710011 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 22798
I20260810 23:32:55.710280 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:32:55.710949 9411 external_mini_cluster.cc:1428] Setting key eede607c33a57b6ff9d79e98dd09db94
I20260810 23:32:55.722867 22905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98: Bootstrap starting.
I20260810 23:32:55.727136 22905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98: Neither blocks nor log segments found. Creating new log.
Aug 10 23:32:55 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:55.728433 22905 log.cc:824] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:55.732390 22905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98: No bootstrap required, opened a new log
I20260810 23:32:55.738737 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:55.713091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55410 (local address 127.9.48.254:39771)
0810 23:32:55.714097 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:55.714116 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:55.714180 (+ 64us) server_negotiation.cc:407] Connection header received
0810 23:32:55.717054 (+ 2874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:55.717086 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:55.717669 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:55.718215 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:55.719525 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:55.721919 (+ 2394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:55.723265 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:55.724434 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:55.724680 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:55.728680 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:55.728695 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:55.728715 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:55.728772 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:55.733568 (+ 4796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:55.734312 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:55.734322 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:55.734332 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:55.734603 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:55.734989 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:55.734996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:55.735000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:55.735859 (+ 859us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:55.736230 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:55.736636 (+ 406us) server_negotiation.cc:299] Negotiation successful
0810 23:32:55.737132 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":392,"threads_started":1}
I20260810 23:32:55.743774 22905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45363f6dee624488ae24ccbfb4478e98" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39771 } }
I20260810 23:32:55.744175 22905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:32:55.744271 22905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45363f6dee624488ae24ccbfb4478e98, State: Initialized, Role: FOLLOWER
I20260810 23:32:55.744927 22905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45363f6dee624488ae24ccbfb4478e98" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39771 } }
I20260810 23:32:55.745190 22905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:32:55.745312 22905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:32:55.745452 22905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:32:55.748332 22905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45363f6dee624488ae24ccbfb4478e98" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39771 } }
I20260810 23:32:55.748958 22905 leader_election.cc:302] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45363f6dee624488ae24ccbfb4478e98; no voters:
I20260810 23:32:55.749552 22905 leader_election.cc:288] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:32:55.749655 22913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:32:55.750922 22913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 1 LEADER]: Becoming Leader. State: Replica: 45363f6dee624488ae24ccbfb4478e98, State: Running, Role: LEADER
I20260810 23:32:55.751503 22913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45363f6dee624488ae24ccbfb4478e98" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39771 } }
I20260810 23:32:55.752115 22905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:55.754498 22915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45363f6dee624488ae24ccbfb4478e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45363f6dee624488ae24ccbfb4478e98" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39771 } } }
I20260810 23:32:55.754887 22915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:55.756980 22921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:55.755873 22916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45363f6dee624488ae24ccbfb4478e98. Latest consensus state: current_term: 1 leader_uuid: "45363f6dee624488ae24ccbfb4478e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45363f6dee624488ae24ccbfb4478e98" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39771 } } }
I20260810 23:32:55.757282 22916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:55.763907 22921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:55.773825 22921 catalog_manager.cc:1347] Generated new cluster ID: 6f478071a99e47fea6709588f35d6d8c
I20260810 23:32:55.773931 22921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:55.793764 22921 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:32:55.795212 22921 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:55.809389 22921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98: Generated new TSK 0
I20260810 23:32:55.810516 22921 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:32:55.889547 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39771
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:56.238469 22944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:56.238785 22944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:56.238952 22944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:56.246356 22944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:56.246510 22944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:56.246680 22944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:32:56.256394 22944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:56.257867 22944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:56.259537 22944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:56.273643 22955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:56.273692 22959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:56.273643 22962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:56.274772 22944 server_base.cc:1034] running on GCE node
I20260810 23:32:56.275394 22944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:56.276364 22944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:56.277601 22944 hybrid_clock.cc:648] HybridClock initialized: now 1786404776277553 us; error 71 us; skew 500 ppm
I20260810 23:32:56.277931 22944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:56 dist-test-slave-04bb krb5kdc[22769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404776, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:56.281109 22944 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:32:56.283398 22944 webserver.cc:460] Webserver started at http://127.9.48.193:40863/ using document root <none> and password file <none>
I20260810 23:32:56.284085 22944 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:56.284224 22944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:56.284631 22944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:56.287528 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "a0683ea13b084a2da34d617e5cb379c5"
format_stamp: "Formatted at 2026-08-10 23:32:56 on dist-test-slave-04bb"
server_key: "863d6101edb0c2e97a50451203367d4b"
server_key_iv: "3a18b745c1dbbe8349827b3b672cbe28"
server_key_version: "encryptionkey@0"
I20260810 23:32:56.288334 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "a0683ea13b084a2da34d617e5cb379c5"
format_stamp: "Formatted at 2026-08-10 23:32:56 on dist-test-slave-04bb"
server_key: "863d6101edb0c2e97a50451203367d4b"
server_key_iv: "3a18b745c1dbbe8349827b3b672cbe28"
server_key_version: "encryptionkey@0"
I20260810 23:32:56.293497 22944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260810 23:32:56.297114 22973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:56.298540 22944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:32:56.298729 22944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "a0683ea13b084a2da34d617e5cb379c5"
format_stamp: "Formatted at 2026-08-10 23:32:56 on dist-test-slave-04bb"
server_key: "863d6101edb0c2e97a50451203367d4b"
server_key_iv: "3a18b745c1dbbe8349827b3b672cbe28"
server_key_version: "encryptionkey@0"
I20260810 23:32:56.298918 22944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:56.317591 22944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:56.321827 22944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:56.322110 22944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:56.323015 22944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:56.324332 22944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:56.324406 22944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:56.324602 22944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:56.324683 22944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 23:32:56.359640 22944 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:42093
I20260810 23:32:56.359678 23150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:42093 every 8 connection(s)
I20260810 23:32:56.360989 22944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:32:56.369145 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 22944
I20260810 23:32:56.369354 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:32:56.370186 9411 external_mini_cluster.cc:1428] Setting key ac174b2bc79ae8c3507a6f38291c5761
Aug 10 23:32:56 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404776, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:56.383144 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:56.363538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:43871 (local address 127.9.48.254:39771)
0810 23:32:56.363994 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:56.364001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:56.365228 (+ 1227us) server_negotiation.cc:407] Connection header received
0810 23:32:56.366967 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:56.366971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:56.367038 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:56.367140 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:56.368636 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:56.369427 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:56.371475 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:56.371965 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:56.372147 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:56.375571 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:56.375581 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:56.375589 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:56.375637 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:56.379608 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:56.380364 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:56.380373 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:56.380380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:56.380572 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:56.380990 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:56.380999 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:56.381005 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:56.381451 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:56.381610 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:56.382327 (+ 717us) server_negotiation.cc:299] Negotiation successful
0810 23:32:56.382620 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":207,"threads_started":1}
I20260810 23:32:56.383669 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:56.364271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:43871)
0810 23:32:56.365078 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:56.365115 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:56.366717 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:56.367345 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:56.367352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:56.367879 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:56.368423 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:56.368440 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:56.369601 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:56.369609 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:56.371328 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:56.371346 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:56.372209 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:56.372218 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:56.372473 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:56.373237 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:56.373264 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:56.375346 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:56.379771 (+ 4425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:56.379778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:56.379790 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:56.380223 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:56.380692 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:56.380695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:56.380698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:56.380872 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:56.381608 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:56.381622 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:56.381926 (+ 304us) client_negotiation.cc:769] Sending connection context
0810 23:32:56.382180 (+ 254us) client_negotiation.cc:240] Negotiation successful
0810 23:32:56.382554 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":344,"threads_started":1}
I20260810 23:32:56.385360 23152 heartbeater.cc:347] Connected to a master server at 127.9.48.254:39771
I20260810 23:32:56.385704 23152 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:56.386477 23152 heartbeater.cc:511] Master 127.9.48.254:39771 requested a full tablet report, sending...
I20260810 23:32:56.389081 22856 ts_manager.cc:194] Registered new tserver with Master: a0683ea13b084a2da34d617e5cb379c5 (127.9.48.193:42093)
I20260810 23:32:56.390496 22856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:43871
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:32:56.432000 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39771
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:56.766367 23163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:56.766657 23163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:56.766815 23163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:56.773844 23163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:56.774000 23163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:56.774174 23163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:32:56.783506 23163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:56.784801 23163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:56.786118 23163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:56.799629 23170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:56.799777 23171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:56.799790 23175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:56.800709 23163 server_base.cc:1034] running on GCE node
I20260810 23:32:56.801344 23163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:56.802300 23163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:56.803563 23163 hybrid_clock.cc:648] HybridClock initialized: now 1786404776803517 us; error 57 us; skew 500 ppm
I20260810 23:32:56.803972 23163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:56 dist-test-slave-04bb krb5kdc[22769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404776, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:56.807233 23163 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:32:56.808729 23163 webserver.cc:460] Webserver started at http://127.9.48.194:32981/ using document root <none> and password file <none>
I20260810 23:32:56.809396 23163 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:56.809558 23163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:56.809917 23163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:56.812811 23163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "16535096a91e4aef986ac8bbe7b324bb"
format_stamp: "Formatted at 2026-08-10 23:32:56 on dist-test-slave-04bb"
server_key: "0f4f485fbe74fb04284c2596f451931f"
server_key_iv: "219f41fe4f543572f808aa9d31405123"
server_key_version: "encryptionkey@0"
I20260810 23:32:56.813524 23163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "16535096a91e4aef986ac8bbe7b324bb"
format_stamp: "Formatted at 2026-08-10 23:32:56 on dist-test-slave-04bb"
server_key: "0f4f485fbe74fb04284c2596f451931f"
server_key_iv: "219f41fe4f543572f808aa9d31405123"
server_key_version: "encryptionkey@0"
I20260810 23:32:56.818356 23163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:32:56.821731 23186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:56.823020 23163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:32:56.823151 23163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "16535096a91e4aef986ac8bbe7b324bb"
format_stamp: "Formatted at 2026-08-10 23:32:56 on dist-test-slave-04bb"
server_key: "0f4f485fbe74fb04284c2596f451931f"
server_key_iv: "219f41fe4f543572f808aa9d31405123"
server_key_version: "encryptionkey@0"
I20260810 23:32:56.823304 23163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:56.841194 23163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:56.845597 23163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:56.845852 23163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:56.846726 23163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:56.848066 23163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:56.848143 23163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:56.848232 23163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:56.848276 23163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:56.883435 23163 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:44111
I20260810 23:32:56.883493 23349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:44111 every 8 connection(s)
I20260810 23:32:56.884852 23163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:32:56.886435 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 23163
I20260810 23:32:56.886627 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:32:56.887075 9411 external_mini_cluster.cc:1428] Setting key 25656275945ed12e02660fbcde7bb935
Aug 10 23:32:56 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404776, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:56.905514 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:56.887575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:39699 (local address 127.9.48.254:39771)
0810 23:32:56.888099 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:56.888103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:56.891127 (+ 3024us) server_negotiation.cc:407] Connection header received
0810 23:32:56.891183 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:56.891186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:56.891251 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:56.891375 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:56.893001 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:56.893738 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:56.895292 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:56.895741 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:56.895892 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:56.899553 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:56.899558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:56.899561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:56.899593 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:56.902343 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:56.903047 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:56.903052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:56.903056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:56.903256 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:56.903704 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:56.903709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:56.903712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:56.903989 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:56.904149 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:56.904889 (+ 740us) server_negotiation.cc:299] Negotiation successful
0810 23:32:56.905112 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":289,"threads_started":1}
I20260810 23:32:56.906479 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:56.888193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:39699)
0810 23:32:56.889102 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:56.889161 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:56.890855 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:56.891627 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:56.891635 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:56.892150 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:56.892823 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:56.892844 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:56.893917 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:56.893921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:56.895139 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:56.895151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:56.895990 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:56.895997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:56.896225 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:56.896950 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:56.896977 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:56.899318 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:56.902481 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:56.902489 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:56.902504 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:56.902912 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:56.903386 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:56.903391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:56.903395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:56.903580 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:56.904122 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:56.904129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:56.904451 (+ 322us) client_negotiation.cc:769] Sending connection context
0810 23:32:56.904783 (+ 332us) client_negotiation.cc:240] Negotiation successful
0810 23:32:56.905285 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":372,"threads_started":1}
I20260810 23:32:56.908018 23350 heartbeater.cc:347] Connected to a master server at 127.9.48.254:39771
I20260810 23:32:56.908334 23350 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:56.909119 23350 heartbeater.cc:511] Master 127.9.48.254:39771 requested a full tablet report, sending...
I20260810 23:32:56.910763 22856 ts_manager.cc:194] Registered new tserver with Master: 16535096a91e4aef986ac8bbe7b324bb (127.9.48.194:44111)
I20260810 23:32:56.912081 22856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:39699
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:32:56.948357 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39771
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:57.285017 23358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:57.285287 23358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:57.285457 23358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:57.292661 23358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:57.292786 23358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:57.292979 23358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:32:57.302618 23358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:57.304054 23358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:57.305473 23358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:57.318606 23366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:57.318542 23368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:57.318640 23365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:57.319801 23358 server_base.cc:1034] running on GCE node
I20260810 23:32:57.320418 23358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:57.321408 23358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:57.322609 23358 hybrid_clock.cc:648] HybridClock initialized: now 1786404777322571 us; error 27 us; skew 500 ppm
I20260810 23:32:57.322930 23358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:57 dist-test-slave-04bb krb5kdc[22769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404777, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:57.326184 23358 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:32:57.327679 23358 webserver.cc:460] Webserver started at http://127.9.48.195:40899/ using document root <none> and password file <none>
I20260810 23:32:57.328434 23358 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:57.328599 23358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:57.328928 23358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:32:57.331810 23358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "1a3c83c431444858bf5ed5795da1abca"
format_stamp: "Formatted at 2026-08-10 23:32:57 on dist-test-slave-04bb"
server_key: "7dbbd6cf4cb7dba7d55a45baa6de4a35"
server_key_iv: "3694cd119d44ff999241aa85810a8531"
server_key_version: "encryptionkey@0"
I20260810 23:32:57.332707 23358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "1a3c83c431444858bf5ed5795da1abca"
format_stamp: "Formatted at 2026-08-10 23:32:57 on dist-test-slave-04bb"
server_key: "7dbbd6cf4cb7dba7d55a45baa6de4a35"
server_key_iv: "3694cd119d44ff999241aa85810a8531"
server_key_version: "encryptionkey@0"
I20260810 23:32:57.337918 23358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 23:32:57.341532 23379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:57.342900 23358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 23:32:57.343096 23358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "1a3c83c431444858bf5ed5795da1abca"
format_stamp: "Formatted at 2026-08-10 23:32:57 on dist-test-slave-04bb"
server_key: "7dbbd6cf4cb7dba7d55a45baa6de4a35"
server_key_iv: "3694cd119d44ff999241aa85810a8531"
server_key_version: "encryptionkey@0"
I20260810 23:32:57.343312 23358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:57.364001 23358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:57.368402 23358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:57.368721 23358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:57.369683 23358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:57.371142 23358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:57.371220 23358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:57.371400 23358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:57.371454 23358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:57.393481 23152 heartbeater.cc:503] Master 127.9.48.254:39771 was elected leader, sending a full tablet report...
I20260810 23:32:57.404378 23358 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:42563
I20260810 23:32:57.404377 23565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:42563 every 8 connection(s)
I20260810 23:32:57.405752 23358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:32:57.414847 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 23358
I20260810 23:32:57.415027 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:32:57.415571 9411 external_mini_cluster.cc:1428] Setting key 5791fce5669df18dff706f908cf4601f
Aug 10 23:32:57 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404777, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:32:57.426618 23568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:57.408441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:54569 (local address 127.9.48.254:39771)
0810 23:32:57.408913 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:57.408918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:57.409972 (+ 1054us) server_negotiation.cc:407] Connection header received
0810 23:32:57.411813 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:57.411818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:57.411912 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:57.412036 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:57.413863 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:57.414703 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:57.416482 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:57.417092 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:57.417230 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:57.421228 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:57.421233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:57.421237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:57.421269 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:57.423748 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:57.424434 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:57.424438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:57.424440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:57.424624 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:57.425016 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:57.425019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:57.425021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:57.425276 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:57.425403 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:57.426063 (+ 660us) server_negotiation.cc:299] Negotiation successful
0810 23:32:57.426224 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
I20260810 23:32:57.427608 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:57.409062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:54569)
0810 23:32:57.409800 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:57.409861 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:57.411534 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:57.412279 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:57.412288 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:57.413078 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:57.413683 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:57.413702 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:57.414860 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:57.414870 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:57.416312 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:57.416329 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:57.417376 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:57.417385 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:57.417692 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:57.418587 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:57.418625 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:57.421040 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:32:57.423876 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:57.423883 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:57.423894 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:57.424325 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:57.424722 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:32:57.424725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:32:57.424728 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:32:57.424912 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:32:57.425426 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:32:57.425432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:32:57.425737 (+ 305us) client_negotiation.cc:769] Sending connection context
0810 23:32:57.425970 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 23:32:57.426407 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":304,"threads_started":1}
I20260810 23:32:57.429054 23566 heartbeater.cc:347] Connected to a master server at 127.9.48.254:39771
I20260810 23:32:57.429320 23566 heartbeater.cc:464] Registering TS with master...
I20260810 23:32:57.430068 23566 heartbeater.cc:511] Master 127.9.48.254:39771 requested a full tablet report, sending...
I20260810 23:32:57.431682 22856 ts_manager.cc:194] Registered new tserver with Master: 1a3c83c431444858bf5ed5795da1abca (127.9.48.195:42563)
I20260810 23:32:57.432468 22856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:54569
I20260810 23:32:57.443795 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.193.keytab.
Entry for principal oryx/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.193.keytab.
WARNING: no policy specified for oryx/127.9.48.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.194.keytab.
Entry for principal oryx/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.194.keytab.
WARNING: no policy specified for oryx/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.195.keytab.
Entry for principal oryx/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.195.keytab.
I20260810 23:32:57.619494 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 22944
Aug 10 23:32:57 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:32:57.636804 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:57.620780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.9.48.193:42093)
0810 23:32:57.621513 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:57.621521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:57.621550 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:32:57.621616 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:57.621623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:57.621849 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:57.621990 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:57.623427 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:57.624456 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:57.625772 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:57.626435 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:57.626538 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:57.630992 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:57.631005 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:57.631022 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:57.631058 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:57.634079 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:57.634894 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:57.634901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:57.634904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:57.635028 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:57.635388 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:57.635391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:57.635393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:57.635733 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:57.635970 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:57.636222 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 23:32:57.636439 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":378,"threads_started":1}
I20260810 23:32:57.638093 23145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:57.747900 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 22944
I20260810 23:32:57.775168 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 23163
Aug 10 23:32:57 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:32:57.791729 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:57.776059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43914 (local address 127.9.48.194:44111)
0810 23:32:57.776661 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:57.776669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:57.776696 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:32:57.776759 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:57.776766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:57.776988 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:57.777161 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:57.778453 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:57.779446 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:57.780833 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:57.781386 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:57.781531 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:57.785808 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:57.785820 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:57.785833 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:57.785865 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:57.788863 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:57.789565 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:57.789572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:57.789575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:57.789696 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:57.790099 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:57.790104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:57.790107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:57.790449 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:57.790788 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:57.791081 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 23:32:57.791350 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":257,"threads_started":1}
I20260810 23:32:57.793102 23345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:57.877688 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 23163
I20260810 23:32:57.902025 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 23358
Aug 10 23:32:57 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:32:57.919317 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:57.903224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.9.48.195:42563)
0810 23:32:57.903769 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:57.903777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:57.903818 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:32:57.903890 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:57.903896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:57.904112 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:57.904297 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:57.905710 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:57.906725 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:57.908056 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:57.908689 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:57.908845 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:57.913269 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:32:57.913280 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:32:57.913295 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:32:57.913336 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:32:57.916426 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:57.917139 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:57.917147 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:57.917150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:57.917275 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:32:57.917674 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:32:57.917677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:32:57.917679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:32:57.918040 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:32:57.918351 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0810 23:32:57.918697 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 23:32:57.918985 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":257,"threads_started":1}
I20260810 23:32:57.920668 23554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:58.005062 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 23358
I20260810 23:32:58.031610 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 22798
I20260810 23:32:58.032899 22898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:32:58.148563 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 22798
I20260810 23:32:58.179133 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:39771
--webserver_interface=127.9.48.254
--webserver_port=42119
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:39771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:32:58.520248 23594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:58.520586 23594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:58.520771 23594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:58.527758 23594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:32:58.527938 23594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:58.528041 23594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:58.528158 23594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:32:58.528249 23594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:32:58.537725 23594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:39771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:39771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=42119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:58.539054 23594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:58.540454 23594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:58.552873 23604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:58.553031 23603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:58.552934 23608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:58.553823 23594 server_base.cc:1034] running on GCE node
I20260810 23:32:58.554682 23594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:58.556099 23594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:58.557376 23594 hybrid_clock.cc:648] HybridClock initialized: now 1786404778557318 us; error 56 us; skew 500 ppm
I20260810 23:32:58.557781 23594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:58 dist-test-slave-04bb krb5kdc[22769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404778, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:58.561022 23594 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:32:58.562727 23594 webserver.cc:460] Webserver started at http://127.9.48.254:42119/ using document root <none> and password file <none>
I20260810 23:32:58.563408 23594 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:58.563571 23594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:58.569067 23594 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:32:58.571787 23619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:58.573061 23594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:58.573237 23594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "45363f6dee624488ae24ccbfb4478e98"
format_stamp: "Formatted at 2026-08-10 23:32:55 on dist-test-slave-04bb"
server_key: "c4f44a56198f5145d3fdb4b2f723f1be"
server_key_iv: "796a000435b37d56d886dbc8e830fc82"
server_key_version: "encryptionkey@0"
I20260810 23:32:58.574172 23594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:58.589067 23594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:58.593551 23594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:58.593842 23594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:58.612962 23594 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:39771
I20260810 23:32:58.612980 23702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:39771 every 8 connection(s)
I20260810 23:32:58.614441 23594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:32:58.616875 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 23594
I20260810 23:32:58.617687 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:42093
--local_ip_for_outbound_sockets=127.9.48.193
--tserver_master_addrs=127.9.48.254:39771
--webserver_port=40863
--webserver_interface=127.9.48.193
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.48.193
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:32:58.621362 23706 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:32:58.624462 23706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98: Bootstrap starting.
I20260810 23:32:58.643532 23706 log.cc:824] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98: Log is configured to *not* fsync() on all Append() calls
I20260810 23:32:58.652635 23706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:32:58.653172 23706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98: Bootstrap complete.
I20260810 23:32:58.662465 23706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45363f6dee624488ae24ccbfb4478e98" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39771 } }
I20260810 23:32:58.663300 23706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45363f6dee624488ae24ccbfb4478e98, State: Initialized, Role: FOLLOWER
I20260810 23:32:58.663893 23706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45363f6dee624488ae24ccbfb4478e98" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39771 } }
I20260810 23:32:58.664155 23706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:32:58.664284 23706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:32:58.664474 23706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:32:58.668159 23706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45363f6dee624488ae24ccbfb4478e98" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39771 } }
I20260810 23:32:58.668732 23706 leader_election.cc:302] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45363f6dee624488ae24ccbfb4478e98; no voters:
I20260810 23:32:58.669399 23706 leader_election.cc:288] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:32:58.669530 23711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:32:58.670775 23711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [term 2 LEADER]: Becoming Leader. State: Replica: 45363f6dee624488ae24ccbfb4478e98, State: Running, Role: LEADER
I20260810 23:32:58.671396 23711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45363f6dee624488ae24ccbfb4478e98" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39771 } }
I20260810 23:32:58.671672 23706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:32:58.675628 23712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "45363f6dee624488ae24ccbfb4478e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45363f6dee624488ae24ccbfb4478e98" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39771 } } }
I20260810 23:32:58.676065 23712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:58.676640 23713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45363f6dee624488ae24ccbfb4478e98. Latest consensus state: current_term: 2 leader_uuid: "45363f6dee624488ae24ccbfb4478e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45363f6dee624488ae24ccbfb4478e98" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39771 } } }
I20260810 23:32:58.677008 23713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98 [sys.catalog]: This master's current role is: LEADER
I20260810 23:32:58.679283 23718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:32:58.689270 23718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:32:58.691052 23718 catalog_manager.cc:1259] Loaded cluster ID: 6f478071a99e47fea6709588f35d6d8c
I20260810 23:32:58.691164 23718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:32:58.696283 23718 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:32:58.698532 23718 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 45363f6dee624488ae24ccbfb4478e98: Loaded TSK: 0
I20260810 23:32:58.700484 23718 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:32:58.983814 23708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:58.984087 23708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:58.984249 23708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:58.991562 23708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:58.991684 23708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:58.991853 23708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:32:59.001817 23708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.193.keytab
--principal=oryx/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:42093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=40863
--tserver_master_addrs=127.9.48.254:39771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:59.003201 23708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:59.004602 23708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:59.019276 23745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:59.019311 23746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:59.020085 23708 server_base.cc:1034] running on GCE node
W20260810 23:32:59.020166 23748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:59.020979 23708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:59.022121 23708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:59.023420 23708 hybrid_clock.cc:648] HybridClock initialized: now 1786404779023362 us; error 67 us; skew 500 ppm
I20260810 23:32:59.023785 23708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:59 dist-test-slave-04bb krb5kdc[22769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:59.031765 23708 init.cc:377] Logged in from keytab as oryx/127.9.48.193@KRBTEST.COM (short username oryx)
I20260810 23:32:59.033674 23708 webserver.cc:460] Webserver started at http://127.9.48.193:40863/ using document root <none> and password file <none>
I20260810 23:32:59.034626 23708 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:59.034777 23708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:59.042069 23708 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:32:59.045459 23759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:59.047055 23708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:32:59.047235 23708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "a0683ea13b084a2da34d617e5cb379c5"
format_stamp: "Formatted at 2026-08-10 23:32:56 on dist-test-slave-04bb"
server_key: "863d6101edb0c2e97a50451203367d4b"
server_key_iv: "3a18b745c1dbbe8349827b3b672cbe28"
server_key_version: "encryptionkey@0"
I20260810 23:32:59.048590 23708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:59.067596 23708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:59.072005 23708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:59.072319 23708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:59.073388 23708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:59.074678 23708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:59.074810 23708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:59.074937 23708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:59.075039 23708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:59.107400 23708 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:42093
I20260810 23:32:59.107468 23925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:42093 every 8 connection(s)
I20260810 23:32:59.108966 23708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:32:59.116773 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 23708
I20260810 23:32:59.117571 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:44111
--local_ip_for_outbound_sockets=127.9.48.194
--tserver_master_addrs=127.9.48.254:39771
--webserver_port=32981
--webserver_interface=127.9.48.194
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.48.194
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:32:59 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:32:59 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:32:59.128924 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:59.111870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:34787)
0810 23:32:59.112774 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:59.112820 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:59.114529 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:59.115808 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:59.115815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:59.116591 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:59.117388 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:59.117417 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.119115 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.119123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:59.121226 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:59.121245 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.122230 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.122240 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:59.122562 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:59.123319 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:59.123352 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:59.127565 (+ 4213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":394,"threads_started":1}
W20260810 23:32:59.129042 23929 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:59.111889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:34787 (local address 127.9.48.254:39771)
0810 23:32:59.112634 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:59.112649 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:59.112938 (+ 289us) server_negotiation.cc:407] Connection header received
0810 23:32:59.115007 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:59.115019 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:59.115358 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:59.115645 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:59.117591 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.118938 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.121366 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.121984 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.122120 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:59.127953 (+ 5833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:34787: BlockingRecv error: recv got EOF from 127.9.48.193:34787 (error 108)
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":291,"threads_started":1}
W20260810 23:32:59.129717 23926 heartbeater.cc:644] Failed to heartbeat to 127.9.48.254:39771 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.48.254:39771: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Aug 10 23:32:59 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:32:59 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:32:59.138115 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:59.130369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:47615)
0810 23:32:59.130566 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:59.130585 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:59.130696 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:59.131023 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:59.131028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:59.131368 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:59.131501 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:59.131509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.132497 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.132504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:59.133450 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:59.133461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.134148 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.134153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:59.134339 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:59.134942 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:59.134958 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:59.137831 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 23:32:59.138236 23929 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:59.130468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:47615 (local address 127.9.48.254:39771)
0810 23:32:59.130626 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:59.130631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:59.130648 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:32:59.130824 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:59.130829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:59.130894 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:59.130997 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:59.131638 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.132354 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.133591 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.134004 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.134137 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:59.138018 (+ 3881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:47615: BlockingRecv error: recv got EOF from 127.9.48.193:47615 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 23:32:59 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:32:59 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:32:59.147579 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:59.138925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:51551)
0810 23:32:59.139119 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:59.139138 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:59.139240 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:59.139699 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:59.139703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:59.139963 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:59.140102 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:59.140110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.141647 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.141650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:59.142527 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:59.142538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.143588 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.143591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:59.143726 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:59.144338 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:59.144354 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:59.147307 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 23:32:59.147909 23926 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:32:59.148048 23929 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:59.139157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:51551 (local address 127.9.48.254:39771)
0810 23:32:59.139365 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:59.139373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:59.139392 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:32:59.139449 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:59.139456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:59.139555 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:59.139678 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:59.140246 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.141503 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.142680 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.143440 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.143585 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:59.147580 (+ 3995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:51551: BlockingRecv error: recv got EOF from 127.9.48.193:51551 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 23:32:59.465101 23931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:59.465452 23931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:59.465649 23931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:59.473829 23931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:59.473994 23931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:59.474196 23931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:32:59.485121 23931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.194.keytab
--principal=oryx/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:44111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=32981
--tserver_master_addrs=127.9.48.254:39771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:59.486581 23931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:59.488055 23931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:59.502436 23940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:59.502575 23939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:59.502581 23942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:59.503424 23931 server_base.cc:1034] running on GCE node
I20260810 23:32:59.504048 23931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:59.505074 23931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:59.506330 23931 hybrid_clock.cc:648] HybridClock initialized: now 1786404779506273 us; error 44 us; skew 500 ppm
I20260810 23:32:59.506666 23931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:59 dist-test-slave-04bb krb5kdc[22769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:59.510228 23931 init.cc:377] Logged in from keytab as oryx/127.9.48.194@KRBTEST.COM (short username oryx)
I20260810 23:32:59.511742 23931 webserver.cc:460] Webserver started at http://127.9.48.194:32981/ using document root <none> and password file <none>
I20260810 23:32:59.512429 23931 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:59.512617 23931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:59.518110 23931 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:32:59.521013 23957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:59.522493 23931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:32:59.522706 23931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "16535096a91e4aef986ac8bbe7b324bb"
format_stamp: "Formatted at 2026-08-10 23:32:56 on dist-test-slave-04bb"
server_key: "0f4f485fbe74fb04284c2596f451931f"
server_key_iv: "219f41fe4f543572f808aa9d31405123"
server_key_version: "encryptionkey@0"
I20260810 23:32:59.523747 23931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:32:59.550081 23931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:32:59.555058 23931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:32:59.555375 23931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:32:59.556495 23931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:32:59.558009 23931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:32:59.558161 23931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:59.558312 23931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:32:59.558421 23931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:59.594950 23931 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:44111
I20260810 23:32:59.595007 24144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:44111 every 8 connection(s)
I20260810 23:32:59.596514 23931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:32:59.597204 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 23931
I20260810 23:32:59.598054 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:42563
--local_ip_for_outbound_sockets=127.9.48.195
--tserver_master_addrs=127.9.48.254:39771
--webserver_port=40899
--webserver_interface=127.9.48.195
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.48.195
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:32:59 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:32:59 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:32:59.615815 23929 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:59.599913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:55957 (local address 127.9.48.254:39771)
0810 23:32:59.600115 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:59.600120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:59.601627 (+ 1507us) server_negotiation.cc:407] Connection header received
0810 23:32:59.603849 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:59.603858 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:59.603985 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:59.604123 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:59.605995 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.606823 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.608690 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.609186 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.609358 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:59.615483 (+ 6125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:55957: BlockingRecv error: recv got EOF from 127.9.48.194:55957 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:32:59.616667 24151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:59.600464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:55957)
0810 23:32:59.601425 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:59.601470 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:59.603521 (+ 2051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:59.604395 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:59.604405 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:59.605100 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:59.605782 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:59.605803 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.607032 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.607038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:59.608463 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:59.608477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.609459 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.609469 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:59.609748 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:59.610544 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:59.610571 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:59.615252 (+ 4681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":356,"threads_started":1}
W20260810 23:32:59.617451 24146 heartbeater.cc:644] Failed to heartbeat to 127.9.48.254:39771 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.48.254:39771: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Aug 10 23:32:59 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:32:59 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:32:59.627672 23929 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:59.618333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:34761 (local address 127.9.48.254:39771)
0810 23:32:59.618552 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:59.618559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:59.618615 (+ 56us) server_negotiation.cc:407] Connection header received
0810 23:32:59.618853 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:59.618860 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:59.618940 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:59.619072 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:59.619827 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.620706 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.622201 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.622741 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.622919 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:59.627343 (+ 4424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:34761: BlockingRecv error: recv got EOF from 127.9.48.194:34761 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:32:59.628289 24151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:59.618252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:34761)
0810 23:32:59.618532 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:59.618556 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:59.618684 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:59.619105 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:59.619111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:59.619509 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:59.619665 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:59.619674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.620861 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.620868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:59.622024 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:59.622038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.622928 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.622935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:59.623163 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:59.623852 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:59.623874 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:59.627946 (+ 4072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146}
Aug 10 23:32:59 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:32:59 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:32:59.638357 24151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:32:59.629216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:60361)
0810 23:32:59.629595 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0810 23:32:59.629613 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:32:59.629757 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:32:59.630151 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:32:59.630157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:32:59.630511 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:32:59.630676 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:59.630686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.631874 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.631884 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:59.633066 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:32:59.633089 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.633900 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.633907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:32:59.634125 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:59.634848 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:32:59.634869 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:32:59.638038 (+ 3169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274}
W20260810 23:32:59.638628 23929 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:32:59.629304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:60361 (local address 127.9.48.254:39771)
0810 23:32:59.629471 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 23:32:59.629476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:32:59.629698 (+ 222us) server_negotiation.cc:407] Connection header received
0810 23:32:59.629910 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:32:59.629917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:32:59.629998 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:32:59.630117 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:32:59.630842 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.631706 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.633253 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:32:59.633727 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:32:59.633892 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:32:59.638268 (+ 4376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:60361: BlockingRecv error: recv got EOF from 127.9.48.194:60361 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 23:32:59.638700 24146 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:32:59.943485 24152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:32:59.943822 24152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:32:59.943991 24152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:32:59.951424 24152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:32:59.951594 24152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:32:59.951766 24152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:32:59.961604 24152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/krb5kdc/oryx_127.9.48.195.keytab
--principal=oryx/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:42563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=40899
--tserver_master_addrs=127.9.48.254:39771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:32:59.962977 24152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:32:59.964360 24152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:32:59.977913 24160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:59.978284 24158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:32:59.978354 24157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:32:59.979084 24152 server_base.cc:1034] running on GCE node
I20260810 23:32:59.979885 24152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:32:59.980897 24152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:32:59.982132 24152 hybrid_clock.cc:648] HybridClock initialized: now 1786404779982071 us; error 39 us; skew 500 ppm
I20260810 23:32:59.982510 24152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:32:59 dist-test-slave-04bb krb5kdc[22769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404779, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:32:59.986197 24152 init.cc:377] Logged in from keytab as oryx/127.9.48.195@KRBTEST.COM (short username oryx)
I20260810 23:32:59.987635 24152 webserver.cc:460] Webserver started at http://127.9.48.195:40899/ using document root <none> and password file <none>
I20260810 23:32:59.988340 24152 fs_manager.cc:359] Metadata directory not provided
I20260810 23:32:59.988464 24152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:32:59.994043 24152 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:32:59.996915 24171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:32:59.998265 24152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:32:59.998443 24152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "1a3c83c431444858bf5ed5795da1abca"
format_stamp: "Formatted at 2026-08-10 23:32:57 on dist-test-slave-04bb"
server_key: "7dbbd6cf4cb7dba7d55a45baa6de4a35"
server_key_iv: "3694cd119d44ff999241aa85810a8531"
server_key_version: "encryptionkey@0"
I20260810 23:32:59.999415 24152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:00.026299 24152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:00.030932 24152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:00.031209 24152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:00.032217 24152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:33:00.033653 24152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:33:00.033744 24152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:00.033906 24152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:33:00.034006 24152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:00.068135 24152 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:42563
I20260810 23:33:00.068225 24351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:42563 every 8 connection(s)
I20260810 23:33:00.069686 24152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:33:00.079704 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 24152
Aug 10 23:33:00 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:00 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:33:00.088471 23929 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:00.072445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:53389 (local address 127.9.48.254:39771)
0810 23:33:00.072736 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:00.072743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:00.074051 (+ 1308us) server_negotiation.cc:407] Connection header received
0810 23:33:00.076082 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:00.076086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:00.076166 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:00.076257 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:00.078014 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.078842 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.080935 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.081820 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.081989 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:00.088150 (+ 6161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:53389: BlockingRecv error: recv got EOF from 127.9.48.195:53389 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260810 23:33:00.089057 24354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:00.073067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:53389)
0810 23:33:00.073865 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:00.073920 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:00.075805 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:00.076494 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:00.076502 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:00.077197 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:00.077852 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:00.077871 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.079005 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.079010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:00.080764 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:00.080783 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.082089 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.082096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:00.082345 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:00.083096 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:00.083136 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:00.087919 (+ 4783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":323,"threads_started":1}
W20260810 23:33:00.089788 24352 heartbeater.cc:644] Failed to heartbeat to 127.9.48.254:39771 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.48.254:39771: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Aug 10 23:33:00 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:00 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:33:00.101238 23929 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:00.090548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:35199 (local address 127.9.48.254:39771)
0810 23:33:00.090747 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:00.090753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:00.090768 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:33:00.090816 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:00.090819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:00.090870 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:00.090970 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:00.091644 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.092858 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.094782 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.095475 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.095577 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:00.100952 (+ 5375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:35199: BlockingRecv error: recv got EOF from 127.9.48.195:35199 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:33:00.101708 24354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:00.090369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:35199)
0810 23:33:00.090547 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:00.090565 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:00.090663 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:00.090998 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:00.091003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:00.091333 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:00.091459 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:00.091467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.093047 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.093055 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:00.094606 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:00.094624 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.095833 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.095841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:00.096032 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:00.097504 (+ 1472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:00.097529 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:00.101441 (+ 3912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 23:33:00.101918 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:00.080796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.9.48.254:39771)
0810 23:33:00.081474 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:00.081483 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:00.082255 (+ 772us) server_negotiation.cc:407] Connection header received
0810 23:33:00.082461 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:00.082467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:00.082541 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:00.082676 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:00.084888 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.086007 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.087948 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.088667 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.088809 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:00.091812 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:00.091827 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:00.091848 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:33:00.091892 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:00.096168 (+ 4276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:00.097423 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:00.097436 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:00.097446 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:00.097708 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:00.099290 (+ 1582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:00.099297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:00.099302 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:00.100031 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:00.100342 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:00.101059 (+ 717us) server_negotiation.cc:299] Negotiation successful
0810 23:33:00.101421 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":321,"threads_started":1}
Aug 10 23:33:00 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:00 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:00.114023 24354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:00.103746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:51367)
0810 23:33:00.104160 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:00.104192 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:00.104304 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:00.105425 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:00.105430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:00.105851 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:00.106067 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:00.106082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.107373 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.107381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:00.108636 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:00.108651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.109636 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.109643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:00.109832 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:00.110426 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:00.110448 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:00.113829 (+ 3381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301}
W20260810 23:33:00.114326 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:00.103823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:51367 (local address 127.9.48.254:39771)
0810 23:33:00.104110 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:00.104117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:00.104285 (+ 168us) server_negotiation.cc:407] Connection header received
0810 23:33:00.104469 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:00.104476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:00.105112 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:00.105221 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:00.106230 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.107179 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.108798 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.109447 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.109574 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:00.114090 (+ 4516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:51367: BlockingRecv error: recv got EOF from 127.9.48.195:51367 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
W20260810 23:33:00.114306 24352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 23:33:00 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:00 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:00.157239 24362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:00.148509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:51923)
0810 23:33:00.149095 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:00.149112 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:00.149269 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:00.149620 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:00.149625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:00.150094 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:00.150325 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:00.150340 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.151366 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.151373 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:00.152350 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:00.152361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.153177 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.153186 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:00.153358 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:00.153938 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:00.153955 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:00.156904 (+ 2949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":311,"threads_started":1}
W20260810 23:33:00.157362 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:00.148673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:51923 (local address 127.9.48.254:39771)
0810 23:33:00.148924 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:00.148931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:00.149202 (+ 271us) server_negotiation.cc:407] Connection header received
0810 23:33:00.149387 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:00.149391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:00.149456 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:00.149542 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:00.150502 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.151209 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.152501 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.152980 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.153113 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:00.157098 (+ 3985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:51923: BlockingRecv error: recv got EOF from 127.9.48.193:51923 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 23:33:00 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:00 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:00.647948 24364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:00.639372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:57769)
0810 23:33:00.639928 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:00.639944 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:00.640062 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:00.640361 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:00.640364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:00.640858 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:00.641051 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:00.641061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.642197 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.642201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:00.643319 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:00.643332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.644137 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.644141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:00.644262 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:00.644909 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:00.644926 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:00.647708 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":300,"threads_started":1}
W20260810 23:33:00.648293 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:00.639439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:57769 (local address 127.9.48.254:39771)
0810 23:33:00.639638 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:00.639643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:00.640063 (+ 420us) server_negotiation.cc:407] Connection header received
0810 23:33:00.640149 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:00.640153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:00.640219 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:00.640310 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:00.641192 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.642038 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.643454 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:00.643976 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:00.644074 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:00.647932 (+ 3858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:57769: BlockingRecv error: recv got EOF from 127.9.48.194:57769 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 10 23:33:01 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:01 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:01.123955 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:01.114921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:44165)
0810 23:33:01.115431 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:01.115447 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:01.115607 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:01.115970 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:01.115974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:01.116468 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:01.116741 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:01.116751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:01.117872 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:01.117880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:01.118852 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:01.118865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:01.119636 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:01.119643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:01.119833 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:01.120451 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:01.120469 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:01.123718 (+ 3249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":278,"threads_started":1}
W20260810 23:33:01.124238 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:01.114959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:44165 (local address 127.9.48.254:39771)
0810 23:33:01.115170 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:01.115178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:01.115532 (+ 354us) server_negotiation.cc:407] Connection header received
0810 23:33:01.115748 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:01.115752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:01.115814 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:01.115917 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:01.116894 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:01.117716 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:01.119019 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:01.119445 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:01.119588 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:01.123956 (+ 4368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:44165: BlockingRecv error: recv got EOF from 127.9.48.195:44165 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 23:33:01 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:01 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:01.166559 24368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:01.158347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:37973)
0810 23:33:01.158981 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:01.158998 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:01.159128 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:01.159442 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:01.159446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:01.159850 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:01.159995 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:01.160004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:01.161001 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:01.161004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:01.161902 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:01.161913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:01.162562 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:01.162567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:01.162723 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:01.163407 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:01.163426 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:01.166302 (+ 2876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":399,"threads_started":1}
W20260810 23:33:01.166653 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:01.158495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:37973 (local address 127.9.48.254:39771)
0810 23:33:01.158757 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:01.158767 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:01.159079 (+ 312us) server_negotiation.cc:407] Connection header received
0810 23:33:01.159244 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:01.159248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:01.159315 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:01.159396 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:01.160115 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:01.160880 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:01.162019 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:01.162393 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:01.162483 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:01.166468 (+ 3985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:37973: BlockingRecv error: recv got EOF from 127.9.48.193:37973 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 23:33:01 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:01 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:01.657826 24369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:01.648988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:56095)
0810 23:33:01.649603 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:01.649619 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:01.649796 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:01.650133 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:01.650136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:01.650558 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:01.650717 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:01.650726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:01.651866 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:01.651871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:01.652890 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:01.652902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:01.653670 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:01.653674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:01.653849 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:01.654583 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:01.654600 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:01.657459 (+ 2859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":333,"threads_started":1}
W20260810 23:33:01.657913 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:01.649023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:56095 (local address 127.9.48.254:39771)
0810 23:33:01.649257 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:01.649266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:01.649741 (+ 475us) server_negotiation.cc:407] Connection header received
0810 23:33:01.649935 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:01.649939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:01.650002 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:01.650081 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:01.650875 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:01.651713 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:01.653071 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:01.653528 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:01.653632 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:01.657673 (+ 4041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:56095: BlockingRecv error: recv got EOF from 127.9.48.194:56095 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 10 23:33:02 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:02 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:02.134177 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:02.125067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:35205)
0810 23:33:02.125653 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:02.125670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:02.125785 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:02.126148 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:02.126151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:02.126580 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:02.126729 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:02.126738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:02.127944 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:02.127948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:02.128981 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:02.128994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:02.129799 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:02.129803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:02.129913 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:02.130641 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:02.130659 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:02.133932 (+ 3273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":302,"threads_started":1}
W20260810 23:33:02.134392 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:02.125215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:35205 (local address 127.9.48.254:39771)
0810 23:33:02.125494 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:02.125502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:02.125756 (+ 254us) server_negotiation.cc:407] Connection header received
0810 23:33:02.125911 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:02.125915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:02.125991 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:02.126108 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:02.126864 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:02.127793 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:02.129135 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:02.129652 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:02.129771 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:02.134144 (+ 4373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:35205: BlockingRecv error: recv got EOF from 127.9.48.195:35205 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 23:33:02 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:02 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:02.176774 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:02.167719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:60133)
0810 23:33:02.168576 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:02.168593 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:02.168706 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:02.169030 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:02.169034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:02.169392 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:02.169594 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:02.169608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:02.170782 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:02.170786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:02.171760 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:02.171774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:02.172478 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:02.172482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:02.172686 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:02.173331 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:02.173348 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:02.176352 (+ 3004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":553,"threads_started":1}
W20260810 23:33:02.176854 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:02.167805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:60133 (local address 127.9.48.254:39771)
0810 23:33:02.168037 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:02.168045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:02.168677 (+ 632us) server_negotiation.cc:407] Connection header received
0810 23:33:02.168813 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:02.168818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:02.168894 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:02.168992 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:02.169770 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:02.170628 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:02.171921 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:02.172346 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:02.172452 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:02.176598 (+ 4146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:60133: BlockingRecv error: recv got EOF from 127.9.48.193:60133 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 10 23:33:02 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:02 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:02.666776 24376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:02.658775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:42027)
0810 23:33:02.659309 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:02.659326 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:02.659448 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:02.659736 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:02.659739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:02.660080 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:02.660261 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:02.660269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:02.661327 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:02.661332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:02.662245 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:02.662257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:02.662905 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:02.662909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:02.663036 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:02.663711 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:02.663729 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:02.666515 (+ 2786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":269,"threads_started":1}
W20260810 23:33:02.666982 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:02.658949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:42027 (local address 127.9.48.254:39771)
0810 23:33:02.659190 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:02.659197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:02.659410 (+ 213us) server_negotiation.cc:407] Connection header received
0810 23:33:02.659536 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:02.659540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:02.659603 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:02.659681 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:02.660386 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:02.661178 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:02.662371 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:02.662779 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:02.662868 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:02.666741 (+ 3873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:42027: BlockingRecv error: recv got EOF from 127.9.48.194:42027 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 10 23:33:03 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:03 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:03.143625 24377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:03.135154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:37351)
0810 23:33:03.135722 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:03.135740 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:03.135904 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:03.136284 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:03.136289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:03.136754 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:03.136953 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:03.136963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:03.138028 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:03.138034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:03.138956 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:03.138969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:03.139717 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:03.139723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:03.139918 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:03.140555 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:03.140573 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:03.143368 (+ 2795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":278,"threads_started":1}
W20260810 23:33:03.143842 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:03.135229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:37351 (local address 127.9.48.254:39771)
0810 23:33:03.135462 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:03.135471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:03.135843 (+ 372us) server_negotiation.cc:407] Connection header received
0810 23:33:03.136043 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:03.136049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:03.136137 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:03.136243 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:03.137126 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:03.137864 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:03.139111 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:03.139564 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:03.139703 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:03.143564 (+ 3861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:37351: BlockingRecv error: recv got EOF from 127.9.48.195:37351 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 10 23:33:03 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:03 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:03.187103 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:03.177758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:56825)
0810 23:33:03.178375 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:03.178394 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:03.178530 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:03.178833 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:03.178836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:03.179290 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:03.179457 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:03.179467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:03.180614 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:03.180640 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:03.182020 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:03.182050 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:03.182783 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:03.182787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:03.182928 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:03.183699 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:03.183720 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:03.186842 (+ 3122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":300,"threads_started":1}
W20260810 23:33:03.187253 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:03.177909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:56825 (local address 127.9.48.254:39771)
0810 23:33:03.178071 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:03.178075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:03.178480 (+ 405us) server_negotiation.cc:407] Connection header received
0810 23:33:03.178635 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:03.178640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:03.178701 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:03.178780 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:03.179591 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:03.180454 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:03.182185 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:03.182635 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:03.182736 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:03.187027 (+ 4291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:56825: BlockingRecv error: recv got EOF from 127.9.48.193:56825 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 23:33:03 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:03 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:03.677637 24379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:03.667935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:39009)
0810 23:33:03.668611 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:03.668633 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:03.668806 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:03.669205 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:03.669211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:03.669669 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:03.669831 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:03.669842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:03.670934 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:03.670942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:03.672058 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:03.672073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:03.672957 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:03.672963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:03.673209 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:03.673975 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:03.674005 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:03.677251 (+ 3246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":372,"threads_started":1}
W20260810 23:33:03.677747 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:03.667983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:39009 (local address 127.9.48.254:39771)
0810 23:33:03.668168 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:03.668173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:03.668732 (+ 559us) server_negotiation.cc:407] Connection header received
0810 23:33:03.668953 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:03.668960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:03.669042 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:03.669166 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:03.669992 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:03.670785 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:03.672219 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:03.672787 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:03.672947 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:03.677460 (+ 4513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:39009: BlockingRecv error: recv got EOF from 127.9.48.194:39009 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 23:33:04 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:04 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:04.157120 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:04.144619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:35775)
0810 23:33:04.145255 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:04.145274 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:04.145436 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:04.145890 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:04.145895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:04.146408 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:04.146576 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:04.146587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:04.147568 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:04.147572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:04.148853 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:04.148870 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:04.149528 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:04.149533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:04.149673 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:04.153338 (+ 3665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:04.153369 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:04.156762 (+ 3393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":307,"threads_started":1}
W20260810 23:33:04.157294 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:04.144700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:35775 (local address 127.9.48.254:39771)
0810 23:33:04.144929 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:04.144939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:04.145377 (+ 438us) server_negotiation.cc:407] Connection header received
0810 23:33:04.145588 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:04.145594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:04.145713 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:04.145842 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:04.146719 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:04.147430 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:04.148996 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:04.149387 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:04.149481 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:04.156997 (+ 7516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:35775: BlockingRecv error: recv got EOF from 127.9.48.195:35775 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 23:33:04 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:04 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:04.196905 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:04.188125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:53607)
0810 23:33:04.188691 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:04.188708 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:04.188824 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:04.189124 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:04.189128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:04.189548 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:04.189722 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:04.189731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:04.190700 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:04.190704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:04.191668 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:04.191681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:04.192531 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:04.192571 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:04.192746 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:04.193644 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:04.193693 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:04.196657 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":270,"threads_started":1}
W20260810 23:33:04.197110 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:04.188214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:53607 (local address 127.9.48.254:39771)
0810 23:33:04.188461 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:04.188468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:04.188797 (+ 329us) server_negotiation.cc:407] Connection header received
0810 23:33:04.188928 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:04.188932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:04.188998 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:04.189078 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:04.189860 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:04.190587 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:04.191799 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:04.192218 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:04.192506 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:04.196878 (+ 4372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:53607: BlockingRecv error: recv got EOF from 127.9.48.193:53607 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 10 23:33:04 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:04 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:04.688004 24383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:04.678869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:51359)
0810 23:33:04.679577 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:04.679599 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:04.679725 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:04.680037 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:04.680041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:04.680480 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:04.680738 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:04.680749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:04.681739 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:04.681743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:04.682760 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:04.682771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:04.683461 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:04.683465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:04.683599 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:04.684233 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:04.684251 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:04.687717 (+ 3466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":368,"threads_started":1}
W20260810 23:33:04.688074 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:04.679088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:51359 (local address 127.9.48.254:39771)
0810 23:33:04.679276 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:04.679280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:04.679674 (+ 394us) server_negotiation.cc:407] Connection header received
0810 23:33:04.679827 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:04.679830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:04.679891 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:04.679976 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:04.680866 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:04.681620 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:04.682881 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:04.683329 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:04.683420 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:04.687905 (+ 4485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:51359: BlockingRecv error: recv got EOF from 127.9.48.194:51359 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 23:33:05 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:05 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:05.166702 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:05.158330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:57165)
0810 23:33:05.158928 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:05.158945 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:05.159060 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:05.159445 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:05.159450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:05.159889 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:05.160062 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:05.160071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:05.161156 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:05.161163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:05.162158 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:05.162170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:05.162913 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:05.162919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:05.163097 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:05.163679 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:05.163698 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:05.166450 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":305,"threads_started":1}
W20260810 23:33:05.166904 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:05.158367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:57165 (local address 127.9.48.254:39771)
0810 23:33:05.158542 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:05.158547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:05.159023 (+ 476us) server_negotiation.cc:407] Connection header received
0810 23:33:05.159158 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:05.159162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:05.159263 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:05.159414 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:05.160221 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:05.161016 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:05.162320 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:05.162744 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:05.162885 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:05.166635 (+ 3750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:57165: BlockingRecv error: recv got EOF from 127.9.48.195:57165 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 23:33:05 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:05 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:05.206730 24385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:05.198015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:45219)
0810 23:33:05.198566 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:05.198583 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:05.198702 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:05.198998 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:05.199001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:05.199404 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:05.199568 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:05.199577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:05.200630 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:05.200634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:05.201531 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:05.201542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:05.202316 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:05.202320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:05.202442 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:05.203062 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:05.203125 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:05.206503 (+ 3378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":327,"threads_started":1}
W20260810 23:33:05.206998 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:05.198056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:45219 (local address 127.9.48.254:39771)
0810 23:33:05.198280 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:05.198290 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:05.198692 (+ 402us) server_negotiation.cc:407] Connection header received
0810 23:33:05.198795 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:05.198799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:05.198864 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:05.198948 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:05.199730 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:05.200444 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:05.201708 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:05.202162 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:05.202273 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:05.206736 (+ 4463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:45219: BlockingRecv error: recv got EOF from 127.9.48.193:45219 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 23:33:05 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:05 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:05.697307 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:05.689123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:44985)
0810 23:33:05.689651 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:05.689668 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:05.689783 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:05.690144 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:05.690147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:05.690540 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:05.690690 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:05.690697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:05.691648 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:05.691652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:05.692644 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:05.692657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:05.693346 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:05.693350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:05.693469 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:05.694054 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:05.694071 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:05.697005 (+ 2934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":320,"threads_started":1}
W20260810 23:33:05.697625 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:05.689150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:44985 (local address 127.9.48.254:39771)
0810 23:33:05.689404 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:05.689414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:05.689758 (+ 344us) server_negotiation.cc:407] Connection header received
0810 23:33:05.689942 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:05.689946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:05.690014 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:05.690101 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:05.690825 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:05.691521 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:05.692798 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:05.693205 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:05.693315 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:05.697293 (+ 3978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:44985: BlockingRecv error: recv got EOF from 127.9.48.194:44985 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 10 23:33:06 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:06 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:06.176512 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:06.167672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:43151)
0810 23:33:06.168241 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:06.168257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:06.168384 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:06.168797 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:06.168801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:06.169181 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:06.169365 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:06.169375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:06.170401 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:06.170405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:06.171293 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:06.171313 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:06.172077 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:06.172082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:06.172267 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:06.172988 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:06.173008 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:06.176274 (+ 3266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":318,"threads_started":1}
W20260810 23:33:06.176733 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:06.167805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:43151 (local address 127.9.48.254:39771)
0810 23:33:06.167993 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:06.167998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:06.168332 (+ 334us) server_negotiation.cc:407] Connection header received
0810 23:33:06.168519 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:06.168544 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:06.168662 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:06.168752 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:06.169502 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:06.170266 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:06.171475 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:06.171928 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:06.172030 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:06.176465 (+ 4435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:43151: BlockingRecv error: recv got EOF from 127.9.48.195:43151 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 23:33:06 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:06 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:06.216488 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:06.207765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:53773)
0810 23:33:06.208370 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:06.208387 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:06.208595 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:06.208951 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:06.208955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:06.209415 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:06.209584 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:06.209593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:06.210817 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:06.210823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:06.211768 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:06.211780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:06.212513 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:06.212519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:06.212720 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:06.213302 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:06.213321 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:06.216119 (+ 2798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":256,"threads_started":1}
W20260810 23:33:06.216637 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:06.207845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:53773 (local address 127.9.48.254:39771)
0810 23:33:06.208060 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:06.208068 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:06.208476 (+ 408us) server_negotiation.cc:407] Connection header received
0810 23:33:06.208731 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:06.208736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:06.208816 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:06.208920 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:06.209733 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:06.210662 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:06.211929 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:06.212356 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:06.212497 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:06.216349 (+ 3852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:53773: BlockingRecv error: recv got EOF from 127.9.48.193:53773 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 10 23:33:06 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:06 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:06.707346 24393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:06.698761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:58259)
0810 23:33:06.699371 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:06.699389 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:06.699557 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:06.699908 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:06.699913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:06.700365 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:06.700589 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:06.700610 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:06.701667 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:06.701673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:06.702603 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:06.702614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:06.703347 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:06.703353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:06.703530 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:06.704129 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:06.704148 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:06.707078 (+ 2930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":282,"threads_started":1}
W20260810 23:33:06.707491 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:06.698838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:58259 (local address 127.9.48.254:39771)
0810 23:33:06.699076 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:06.699080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:06.699487 (+ 407us) server_negotiation.cc:407] Connection header received
0810 23:33:06.699698 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:06.699703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:06.699770 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:06.699880 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:06.700744 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:06.701526 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:06.702748 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:06.703157 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:06.703340 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:06.707295 (+ 3955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:58259: BlockingRecv error: recv got EOF from 127.9.48.194:58259 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Aug 10 23:33:07 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:07 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:07.186738 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:07.177601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:51341)
0810 23:33:07.178292 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:07.178308 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:07.178422 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:07.178858 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:07.178865 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:07.179413 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:07.179593 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:07.179603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:07.180807 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:07.180814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:07.181858 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:07.181872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:07.182682 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:07.182688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:07.182867 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:07.183422 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:07.183440 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:07.186478 (+ 3038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":298,"threads_started":1}
W20260810 23:33:07.186949 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:07.177680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:51341 (local address 127.9.48.254:39771)
0810 23:33:07.177905 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:07.177913 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:07.178369 (+ 456us) server_negotiation.cc:407] Connection header received
0810 23:33:07.178584 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:07.178590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:07.178688 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:07.178802 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:07.179756 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:07.180516 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:07.182013 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:07.182506 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:07.182651 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:07.186703 (+ 4052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:51341: BlockingRecv error: recv got EOF from 127.9.48.195:51341 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 23:33:07 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:07 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:07.226481 24395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:07.217532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:49843)
0810 23:33:07.218245 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:07.218264 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:07.218390 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:07.218737 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:07.218742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:07.219257 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:07.219460 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:07.219471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:07.220611 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:07.220616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:07.221652 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:07.221665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:07.222376 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:07.222381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:07.222504 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:07.223093 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:07.223111 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:07.226243 (+ 3132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":409,"threads_started":1}
W20260810 23:33:07.226639 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:07.217678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:49843 (local address 127.9.48.254:39771)
0810 23:33:07.217864 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:07.217869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:07.218340 (+ 471us) server_negotiation.cc:407] Connection header received
0810 23:33:07.218488 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:07.218492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:07.218551 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:07.218631 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:07.219610 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:07.220344 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:07.221791 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:07.222213 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:07.222310 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:07.226394 (+ 4084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:49843: BlockingRecv error: recv got EOF from 127.9.48.193:49843 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 23:33:07 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:07 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:07.722121 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:07.708377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:55919)
0810 23:33:07.709002 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:07.709020 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:07.709142 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:07.709486 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:07.709490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:07.709914 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:07.710095 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:07.710104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:07.711349 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:07.711353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:07.712385 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:07.712399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:07.713484 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:07.713492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:07.713681 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:07.718754 (+ 5073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:07.718774 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:07.721786 (+ 3012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":341,"threads_started":1}
W20260810 23:33:07.722306 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:07.708503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:55919 (local address 127.9.48.254:39771)
0810 23:33:07.708685 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:07.708690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:07.709111 (+ 421us) server_negotiation.cc:407] Connection header received
0810 23:33:07.709258 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:07.709262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:07.709332 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:07.709416 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:07.710250 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:07.711203 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:07.712574 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:07.713277 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:07.713411 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:07.722053 (+ 8642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:55919: BlockingRecv error: recv got EOF from 127.9.48.194:55919 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 10 23:33:08 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:08 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:08.196204 24397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:08.187808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:58825)
0810 23:33:08.188430 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:08.188448 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:08.188625 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:08.188972 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:08.188975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:08.189398 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:08.189556 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:08.189565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:08.190518 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:08.190522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:08.191457 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:08.191468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:08.192211 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:08.192217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:08.192360 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:08.193090 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:08.193109 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:08.195944 (+ 2835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":292,"threads_started":1}
W20260810 23:33:08.196360 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:08.187876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:58825 (local address 127.9.48.254:39771)
0810 23:33:08.188102 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:08.188109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:08.188540 (+ 431us) server_negotiation.cc:407] Connection header received
0810 23:33:08.188753 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:08.188757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:08.188844 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:08.188926 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:08.189678 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:08.190378 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:08.191576 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:08.191978 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:08.192069 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:08.196152 (+ 4083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:58825: BlockingRecv error: recv got EOF from 127.9.48.195:58825 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 23:33:08 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:08 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:08.236693 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:08.227579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:45427)
0810 23:33:08.228198 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:08.228216 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:08.228335 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:08.228745 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:08.228749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:08.229157 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:08.229328 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:08.229338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:08.230392 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:08.230396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:08.231393 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:08.231406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:08.232113 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:08.232117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:08.232244 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:08.232947 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:08.232965 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:08.236367 (+ 3402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":322,"threads_started":1}
W20260810 23:33:08.236899 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:08.227667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:45427 (local address 127.9.48.254:39771)
0810 23:33:08.227930 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:08.227939 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:08.228302 (+ 363us) server_negotiation.cc:407] Connection header received
0810 23:33:08.228434 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:08.228438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:08.228503 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:08.228644 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:08.229469 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:08.230224 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:08.231529 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:08.231976 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:08.232077 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:08.236598 (+ 4521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:45427: BlockingRecv error: recv got EOF from 127.9.48.193:45427 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 23:33:08 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:08 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:08.732197 24400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:08.723102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:44563)
0810 23:33:08.723754 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:08.723770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:08.723889 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:08.724188 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:08.724191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:08.724614 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:08.724811 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:08.724820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:08.725805 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:08.725809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:08.726913 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:08.726927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:08.727930 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:08.727936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:08.728127 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:08.728855 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:08.728873 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:08.731829 (+ 2956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":349,"threads_started":1}
W20260810 23:33:08.732267 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:08.723286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:44563 (local address 127.9.48.254:39771)
0810 23:33:08.723466 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:08.723471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:08.723845 (+ 374us) server_negotiation.cc:407] Connection header received
0810 23:33:08.723988 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:08.723992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:08.724061 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:08.724144 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:08.724940 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:08.725668 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:08.727062 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:08.727738 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:08.727908 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:08.732041 (+ 4133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:44563: BlockingRecv error: recv got EOF from 127.9.48.194:44563 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 23:33:09 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:09 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:09.209666 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:09.197217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:51497)
0810 23:33:09.197762 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:09.197779 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:09.197969 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:09.198328 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:09.198332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:09.198791 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:09.198968 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:09.198977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:09.200066 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:09.200073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:09.201066 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:09.201077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:09.201836 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:09.201842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:09.202034 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:09.205952 (+ 3918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:09.205971 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:09.209291 (+ 3320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":305,"threads_started":1}
W20260810 23:33:09.209775 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:09.197298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:51497 (local address 127.9.48.254:39771)
0810 23:33:09.197526 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:09.197534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:09.197892 (+ 358us) server_negotiation.cc:407] Connection header received
0810 23:33:09.198102 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:09.198106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:09.198185 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:09.198292 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:09.199124 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:09.199928 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:09.201214 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:09.201665 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:09.201824 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:09.209513 (+ 7689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:51497: BlockingRecv error: recv got EOF from 127.9.48.195:51497 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 23:33:09 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:09 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:09.246168 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:09.237655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:35929)
0810 23:33:09.238141 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:09.238158 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:09.238304 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:09.238604 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:09.238607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:09.238977 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:09.239124 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:09.239132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:09.240149 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:09.240153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:09.241181 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:09.241202 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:09.241861 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:09.241865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:09.241989 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:09.242604 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:09.242621 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:09.245840 (+ 3219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":266,"threads_started":1}
W20260810 23:33:09.246259 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:09.237724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:35929 (local address 127.9.48.254:39771)
0810 23:33:09.237982 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:09.237989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:09.238267 (+ 278us) server_negotiation.cc:407] Connection header received
0810 23:33:09.238411 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:09.238415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:09.238479 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:09.238560 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:09.239265 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:09.240004 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:09.241322 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:09.241722 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:09.241817 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:09.246031 (+ 4214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:35929: BlockingRecv error: recv got EOF from 127.9.48.193:35929 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 23:33:09 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:09 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:09.741775 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:09.733192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:38649)
0810 23:33:09.733781 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:09.733799 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:09.733924 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:09.734284 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:09.734288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:09.734734 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:09.734885 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:09.734894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:09.736080 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:09.736088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:09.737107 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:09.737120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:09.737863 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:09.737870 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:09.738056 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:09.738644 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:09.738663 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:09.741547 (+ 2884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":347,"threads_started":1}
W20260810 23:33:09.742087 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:09.733310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:38649 (local address 127.9.48.254:39771)
0810 23:33:09.733485 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:09.733490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:09.733908 (+ 418us) server_negotiation.cc:407] Connection header received
0810 23:33:09.734028 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:09.734032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:09.734096 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:09.734237 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:09.735074 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:09.735909 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:09.737267 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:09.737704 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:09.737844 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:09.741777 (+ 3933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:38649: BlockingRecv error: recv got EOF from 127.9.48.194:38649 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 23:33:10 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:10 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:10.219627 24411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:10.210727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:33509)
0810 23:33:10.211370 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:10.211390 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:10.211521 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:10.211861 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:10.211865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:10.212443 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:10.212676 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:10.212694 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:10.213964 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:10.213970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:10.214964 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:10.214977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:10.215754 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:10.215759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:10.215895 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:10.216599 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:10.216618 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:10.219403 (+ 2785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":379,"threads_started":1}
W20260810 23:33:10.219920 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:10.211037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:33509 (local address 127.9.48.254:39771)
0810 23:33:10.211261 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:10.211268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:10.211478 (+ 210us) server_negotiation.cc:407] Connection header received
0810 23:33:10.211624 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:10.211628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:10.211719 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:10.211804 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:10.212873 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:10.213803 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:10.215118 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:10.215566 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:10.215683 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:10.219616 (+ 3933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:33509: BlockingRecv error: recv got EOF from 127.9.48.195:33509 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 10 23:33:10 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:10 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:10.255721 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:10.247035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:52223)
0810 23:33:10.247562 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:10.247580 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:10.247772 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:10.248156 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:10.248161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:10.248725 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:10.248912 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:10.248921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:10.249952 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:10.249959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:10.250858 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:10.250870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:10.251612 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:10.251619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:10.251791 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:10.252368 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:10.252387 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:10.255481 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":285,"threads_started":1}
W20260810 23:33:10.255954 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:10.247111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:52223 (local address 127.9.48.254:39771)
0810 23:33:10.247336 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:10.247344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:10.247694 (+ 350us) server_negotiation.cc:407] Connection header received
0810 23:33:10.247934 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:10.247944 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:10.248007 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:10.248150 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:10.249065 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:10.249816 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:10.251005 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:10.251441 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:10.251574 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:10.255718 (+ 4144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:52223: BlockingRecv error: recv got EOF from 127.9.48.193:52223 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 23:33:10 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:10 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:10.751343 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:10.742666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:47161)
0810 23:33:10.743155 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:10.743171 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:10.743288 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:10.743596 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:10.743599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:10.744030 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:10.744215 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:10.744223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:10.745314 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:10.745318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:10.746206 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:10.746217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:10.746897 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:10.746901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:10.747015 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:10.747597 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:10.747613 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:10.750972 (+ 3359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":265,"threads_started":1}
W20260810 23:33:10.751600 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:10.742773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:47161 (local address 127.9.48.254:39771)
0810 23:33:10.743035 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:10.743042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:10.743251 (+ 209us) server_negotiation.cc:407] Connection header received
0810 23:33:10.743392 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:10.743397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:10.743465 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:10.743550 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:10.744361 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:10.745201 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:10.746340 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:10.746764 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:10.746860 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:10.751285 (+ 4425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:47161: BlockingRecv error: recv got EOF from 127.9.48.194:47161 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Aug 10 23:33:11 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:11 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:11.230165 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:11.220576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:47483)
0810 23:33:11.221208 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:11.221229 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:11.221414 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:11.221830 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:11.221836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:11.222333 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:11.222531 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:11.222542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:11.223648 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:11.223658 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:11.224641 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:11.224654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:11.225446 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:11.225453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:11.225648 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:11.226297 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:11.226317 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:11.229755 (+ 3438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":304,"threads_started":1}
W20260810 23:33:11.230355 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:11.220656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:47483 (local address 127.9.48.254:39771)
0810 23:33:11.220921 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:11.220932 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:11.221330 (+ 398us) server_negotiation.cc:407] Connection header received
0810 23:33:11.221557 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:11.221563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:11.221665 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:11.221787 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:11.222702 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:11.223488 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:11.224799 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:11.225269 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:11.225423 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:11.230050 (+ 4627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:47483: BlockingRecv error: recv got EOF from 127.9.48.195:47483 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 10 23:33:11 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:11 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:11.265923 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:11.256626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:53149)
0810 23:33:11.257175 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:11.257192 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:11.257365 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:11.257761 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:11.257768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:11.258227 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:11.258392 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:11.258402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:11.259449 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:11.259461 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:11.260421 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:11.260433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:11.261264 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:11.261271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:11.261501 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:11.262219 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:11.262240 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:11.265521 (+ 3281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":306,"threads_started":1}
W20260810 23:33:11.266053 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:11.256709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:53149 (local address 127.9.48.254:39771)
0810 23:33:11.256923 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:11.256932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:11.257283 (+ 351us) server_negotiation.cc:407] Connection header received
0810 23:33:11.257518 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:11.257524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:11.257602 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:11.257710 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:11.258548 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:11.259295 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:11.260615 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:11.261098 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:11.261250 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:11.265769 (+ 4519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:53149: BlockingRecv error: recv got EOF from 127.9.48.193:53149 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 23:33:11 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:11 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:11.761700 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:11.752464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:48783)
0810 23:33:11.753090 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:11.753107 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:11.753273 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:11.753637 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:11.753640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:11.754132 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:11.754328 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:11.754343 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:11.755546 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:11.755550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:11.756746 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:11.756758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:11.757640 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:11.757646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:11.757842 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:11.758514 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:11.758531 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:11.761455 (+ 2924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":360,"threads_started":1}
W20260810 23:33:11.761973 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:11.752596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:48783 (local address 127.9.48.254:39771)
0810 23:33:11.752864 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:11.752871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:11.753210 (+ 339us) server_negotiation.cc:407] Connection header received
0810 23:33:11.753410 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:11.753415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:11.753482 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:11.753587 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:11.754465 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:11.755401 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:11.756902 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:11.757481 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:11.757618 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:11.761653 (+ 4035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:48783: BlockingRecv error: recv got EOF from 127.9.48.194:48783 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 23:33:12 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:12 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:12.241016 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:12.231177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:48111)
0810 23:33:12.231807 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:12.231829 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:12.232016 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:12.232454 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:12.232460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:12.232922 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:12.233081 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:12.233090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:12.234799 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:12.234803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:12.235769 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:12.235784 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:12.237000 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:12.237005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:12.237138 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:12.237859 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:12.237876 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:12.240777 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":310,"threads_started":1}
W20260810 23:33:12.241374 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:12.231197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:48111 (local address 127.9.48.254:39771)
0810 23:33:12.231385 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:12.231393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:12.231906 (+ 513us) server_negotiation.cc:407] Connection header received
0810 23:33:12.232166 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:12.232175 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:12.232275 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:12.232409 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:12.233235 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:12.234625 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:12.235940 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:12.236856 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:12.237018 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:12.241064 (+ 4046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:48111: BlockingRecv error: recv got EOF from 127.9.48.195:48111 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 23:33:12 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:12 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:12.277112 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:12.267013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:36621)
0810 23:33:12.267604 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:12.267626 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:12.267951 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:12.268419 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:12.268425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:12.268999 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:12.269246 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:12.269255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:12.270419 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:12.270423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:12.271372 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:12.271407 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:12.272652 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:12.272658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:12.272780 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:12.273541 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:12.273558 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:12.276799 (+ 3241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":283,"threads_started":1}
W20260810 23:33:12.277570 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:12.267044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:36621 (local address 127.9.48.254:39771)
0810 23:33:12.267244 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:12.267253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:12.267728 (+ 475us) server_negotiation.cc:407] Connection header received
0810 23:33:12.268107 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:12.268115 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:12.268227 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:12.268342 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:12.269441 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:12.270249 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:12.271552 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:12.272408 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:12.272622 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:12.277088 (+ 4466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:36621: BlockingRecv error: recv got EOF from 127.9.48.193:36621 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 23:33:12 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:12 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:12.771453 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:12.762586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:44983)
0810 23:33:12.763220 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:12.763236 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:12.763402 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:12.763771 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:12.763780 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:12.764222 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:12.764393 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:12.764402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:12.765506 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:12.765512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:12.766463 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:12.766476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:12.767275 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:12.767281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:12.767449 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:12.768123 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:12.768143 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:12.771104 (+ 2961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":348,"threads_started":1}
W20260810 23:33:12.771572 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:12.762825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:44983 (local address 127.9.48.254:39771)
0810 23:33:12.763086 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:12.763095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:12.763329 (+ 234us) server_negotiation.cc:407] Connection header received
0810 23:33:12.763533 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:12.763538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:12.763615 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:12.763727 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:12.764570 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:12.765356 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:12.766616 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:12.767075 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:12.767230 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:12.771330 (+ 4100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:44983: BlockingRecv error: recv got EOF from 127.9.48.194:44983 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Aug 10 23:33:13 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:13 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:13.250442 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:13.242022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:35989)
0810 23:33:13.242629 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:13.242647 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:13.242771 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:13.243076 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:13.243080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:13.243499 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:13.243660 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:13.243669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:13.244667 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:13.244671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:13.245605 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:13.245616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:13.246333 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:13.246338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:13.246454 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:13.247036 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:13.247053 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:13.250032 (+ 2979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":337,"threads_started":1}
W20260810 23:33:13.250541 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:13.242119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:35989 (local address 127.9.48.254:39771)
0810 23:33:13.242320 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:13.242324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:13.242726 (+ 402us) server_negotiation.cc:407] Connection header received
0810 23:33:13.242874 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:13.242877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:13.242941 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:13.243024 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:13.243790 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:13.244494 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:13.245731 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:13.246155 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:13.246244 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:13.250280 (+ 4036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:35989: BlockingRecv error: recv got EOF from 127.9.48.195:35989 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 23:33:13 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:13 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:13.286937 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:13.278242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:37373)
0810 23:33:13.278794 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:13.278821 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:13.278937 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:13.279325 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:13.279329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:13.279683 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:13.279881 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:13.279890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:13.281071 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:13.281075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:13.282084 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:13.282097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:13.282784 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:13.282788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:13.282914 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:13.283602 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:13.283620 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:13.286573 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":290,"threads_started":1}
W20260810 23:33:13.287184 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:13.278307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:37373 (local address 127.9.48.254:39771)
0810 23:33:13.278622 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:13.278627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:13.278902 (+ 275us) server_negotiation.cc:407] Connection header received
0810 23:33:13.279085 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:13.279094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:13.279184 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:13.279280 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:13.280022 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:13.280948 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:13.282217 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:13.282637 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:13.282760 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:13.286816 (+ 4056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:37373: BlockingRecv error: recv got EOF from 127.9.48.193:37373 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Aug 10 23:33:13 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:13 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:13.781879 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:13.772562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:56445)
0810 23:33:13.773044 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:13.773061 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:13.773179 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:13.773516 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:13.773520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:13.773926 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:13.774111 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:13.774120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:13.775468 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:13.775473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:13.776727 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:13.776741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:13.777794 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:13.777800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:13.777985 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:13.778676 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:13.778694 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:13.781606 (+ 2912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":256,"threads_started":1}
W20260810 23:33:13.782130 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:13.772652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:56445 (local address 127.9.48.254:39771)
0810 23:33:13.772887 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:13.772894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:13.773147 (+ 253us) server_negotiation.cc:407] Connection header received
0810 23:33:13.773318 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:13.773323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:13.773387 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:13.773471 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:13.774315 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:13.775226 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:13.776926 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:13.777653 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:13.777776 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:13.781833 (+ 4057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:56445: BlockingRecv error: recv got EOF from 127.9.48.194:56445 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 23:33:14 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:14 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.195@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:14.265695 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:14.251636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.195:42641)
0810 23:33:14.252257 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:14.252274 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:14.252438 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:14.252898 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:14.252903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:14.253382 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:14.253567 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:14.253576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:14.254726 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:14.254732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:14.255666 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:14.255678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:14.256446 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:14.256452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:14.256683 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:14.262078 (+ 5395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:14.262098 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:14.265402 (+ 3304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":324,"threads_started":1}
W20260810 23:33:14.265883 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:14.251749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:42641 (local address 127.9.48.254:39771)
0810 23:33:14.251988 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:14.251997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:14.252365 (+ 368us) server_negotiation.cc:407] Connection header received
0810 23:33:14.252617 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:14.252623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:14.252726 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:14.252809 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:14.253713 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:14.254567 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:14.255817 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:14.256294 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:14.256431 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:14.265620 (+ 9189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.195:42641: BlockingRecv error: recv got EOF from 127.9.48.195:42641 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 10 23:33:14 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:14 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.193@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:14.296741 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:14.288304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.193:54671)
0810 23:33:14.288894 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:14.288909 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:14.289048 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:14.289414 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:14.289419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:14.289884 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:14.290094 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:14.290104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:14.291095 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:14.291101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:14.292040 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:14.292050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:14.292786 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:14.292792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:14.292994 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:14.293553 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:14.293569 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:14.296337 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":336,"threads_started":1}
W20260810 23:33:14.296835 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:14.288382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:54671 (local address 127.9.48.254:39771)
0810 23:33:14.288600 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:14.288606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:14.288996 (+ 390us) server_negotiation.cc:407] Connection header received
0810 23:33:14.289189 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:14.289193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:14.289257 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:14.289378 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:14.290248 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:14.290943 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:14.292179 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:14.292638 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:14.292775 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:14.296579 (+ 3804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.193:54671: BlockingRecv error: recv got EOF from 127.9.48.193:54671 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 23:33:14 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:33:14 dist-test-slave-04bb krb5kdc[22769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.48.194@KRBTEST.COM for oryx/127.9.48.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:33:14.791561 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:14.782942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39771 (local address 127.9.48.194:54697)
0810 23:33:14.783496 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:14.783513 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:14.783627 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:14.783944 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:14.783947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:14.784414 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:14.784758 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:14.784771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:14.785964 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:14.785970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:14.786888 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:14.786899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:14.787612 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:14.787618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:14.787789 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:14.788359 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:14.788376 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:14.791191 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.48.254:39771: Server oryx/127.9.48.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":315,"threads_started":1}
W20260810 23:33:14.791642 24359 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:14.783037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:54697 (local address 127.9.48.254:39771)
0810 23:33:14.783205 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:14.783209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:14.783587 (+ 378us) server_negotiation.cc:407] Connection header received
0810 23:33:14.783723 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:14.783727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:14.783821 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:14.783899 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:14.784945 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:14.785810 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:14.787029 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:14.787466 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:14.787612 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:14.791394 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.48.194:54697: BlockingRecv error: recv got EOF from 127.9.48.194:54697 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:33:15.080781 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 23708
I20260810 23:33:15.094089 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:15.081982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.9.48.193:42093)
0810 23:33:15.082709 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:15.082717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:15.082744 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:33:15.082806 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:15.082813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:15.083039 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:15.083186 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:15.084973 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:15.086194 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:15.087405 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:15.087962 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:15.088060 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:15.090870 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:15.090879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:15.090893 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:33:15.090925 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:15.093581 (+ 2656us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.48.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 23:33:15.093843 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58084: Request ticket server kudu/127.9.48.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":419,"threads_started":1}
W20260810 23:33:15.094177 24438 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58084: Request ticket server kudu/127.9.48.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 23:33:15.094440 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 23708: Not authorized: Client connection negotiation failed: client connection to 127.9.48.193:42093: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.48.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 23:33:15.094614 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 23708
I20260810 23:33:15.122747 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 23931
I20260810 23:33:15.135624 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:15.123881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39888 (local address 127.9.48.194:44111)
0810 23:33:15.124457 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:15.124465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:15.124496 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:33:15.124614 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:15.124621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:15.124862 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:15.125015 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:15.126282 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:15.127360 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:15.128999 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:15.129559 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:15.129659 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:15.132406 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:15.132414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:15.132428 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:33:15.132463 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:15.135133 (+ 2670us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.48.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 23:33:15.135388 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39888: Request ticket server kudu/127.9.48.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":270,"threads_started":1}
W20260810 23:33:15.135767 24439 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39888: Request ticket server kudu/127.9.48.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 23:33:15.135880 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 23931: Not authorized: Client connection negotiation failed: client connection to 127.9.48.194:44111: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.48.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 23:33:15.136060 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 23931
I20260810 23:33:15.166600 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 24152
I20260810 23:33:15.179018 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:15.167632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.9.48.195:42563)
0810 23:33:15.168250 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:15.168258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:15.168287 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:33:15.168352 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:15.168358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:15.168656 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:15.168824 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:15.170201 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:15.171262 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:15.172511 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:15.173267 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:15.173374 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:15.175841 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:15.175850 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:15.175864 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:33:15.175905 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:15.178474 (+ 2569us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.48.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 23:33:15.178728 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50354: Request ticket server kudu/127.9.48.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":319,"threads_started":1}
W20260810 23:33:15.179134 24440 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50354: Request ticket server kudu/127.9.48.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 23:33:15.179214 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 24152: Not authorized: Client connection negotiation failed: client connection to 127.9.48.195:42563: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.48.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 23:33:15.179399 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 24152
I20260810 23:33:15.206496 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 23594
I20260810 23:33:15.207907 23696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:33:15.327479 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 23594
2026-08-10T23:33:15Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21673 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T23:33:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:33:15Z Disabled control of system clock
I20260810 23:33:15.426385 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:34445
--webserver_interface=127.9.48.254
--webserver_port=0
--builtin_ntp_servers=127.9.48.212:43125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:34445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 23:33:15.755189 24447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:15.755462 24447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:15.755542 24447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:15.762482 24447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:33:15.762579 24447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:33:15.762617 24447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:15.762674 24447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:33:15.762719 24447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:33:15.771495 24447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:43125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:34445
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:34445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:33:15.772783 24447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:15.774560 24447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:15.786834 24454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:15.786913 24456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:15.786806 24459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:15.787290 24447 server_base.cc:1034] running on GCE node
I20260810 23:33:15.788268 24447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:15.789590 24447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:15.790799 24447 hybrid_clock.cc:648] HybridClock initialized: now 1786404795790746 us; error 38 us; skew 500 ppm
I20260810 23:33:15.791158 24447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:15.793411 24447 webserver.cc:460] Webserver started at http://127.9.48.254:40851/ using document root <none> and password file <none>
I20260810 23:33:15.794102 24447 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:15.794229 24447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:15.794579 24447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:15.797518 24447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "0e7f25dd5ba24a82b7b219d3a7cf2351"
format_stamp: "Formatted at 2026-08-10 23:33:15 on dist-test-slave-04bb"
server_key: "6edb31c95d3173bfdbb95ffe9b53dc01"
server_key_iv: "ecdf0d319debf7c8a27c06cb3a0dc237"
server_key_version: "encryptionkey@0"
I20260810 23:33:15.798364 24447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "0e7f25dd5ba24a82b7b219d3a7cf2351"
format_stamp: "Formatted at 2026-08-10 23:33:15 on dist-test-slave-04bb"
server_key: "6edb31c95d3173bfdbb95ffe9b53dc01"
server_key_iv: "ecdf0d319debf7c8a27c06cb3a0dc237"
server_key_version: "encryptionkey@0"
I20260810 23:33:15.803480 24447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 23:33:15.806985 24467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:15.808470 24447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:33:15.808689 24447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "0e7f25dd5ba24a82b7b219d3a7cf2351"
format_stamp: "Formatted at 2026-08-10 23:33:15 on dist-test-slave-04bb"
server_key: "6edb31c95d3173bfdbb95ffe9b53dc01"
server_key_iv: "ecdf0d319debf7c8a27c06cb3a0dc237"
server_key_version: "encryptionkey@0"
I20260810 23:33:15.808923 24447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:15.825738 24447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:15.826561 24447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:15.826855 24447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:15.845573 24447 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:34445
I20260810 23:33:15.845587 24555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:34445 every 8 connection(s)
I20260810 23:33:15.846954 24447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:33:15.849077 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 24447
I20260810 23:33:15.849298 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:33:15.850201 9411 external_mini_cluster.cc:1428] Setting key 44f11be3771b5995f19375d4b179f62b
I20260810 23:33:15.851678 24556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:15.863807 24556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351: Bootstrap starting.
I20260810 23:33:15.863859 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:15.852378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44056 (local address 127.9.48.254:34445)
0810 23:33:15.853368 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:15.853379 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:15.853423 (+ 44us) server_negotiation.cc:407] Connection header received
0810 23:33:15.855024 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:15.855045 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:15.855503 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:15.855929 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:15.856571 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:15.857962 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:15.859346 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:15.860003 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:15.860209 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:15.861551 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:15.861559 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:15.861572 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:33:15.862140 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:15.862211 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:33:15.862235 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:15.862367 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:15.862603 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 23:33:15.862922 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":334,"threads_started":1}
I20260810 23:33:15.867486 24556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:15.868520 24556 log.cc:824] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351: Log is configured to *not* fsync() on all Append() calls
I20260810 23:33:15.871212 24556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351: No bootstrap required, opened a new log
I20260810 23:33:15.878741 24556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7f25dd5ba24a82b7b219d3a7cf2351" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 34445 } }
I20260810 23:33:15.879117 24556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:15.879230 24556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7f25dd5ba24a82b7b219d3a7cf2351, State: Initialized, Role: FOLLOWER
I20260810 23:33:15.879866 24556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7f25dd5ba24a82b7b219d3a7cf2351" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 34445 } }
I20260810 23:33:15.880115 24556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:33:15.880292 24556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:33:15.880505 24556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:15.882893 24556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7f25dd5ba24a82b7b219d3a7cf2351" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 34445 } }
I20260810 23:33:15.883348 24556 leader_election.cc:302] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e7f25dd5ba24a82b7b219d3a7cf2351; no voters:
I20260810 23:33:15.883911 24556 leader_election.cc:288] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:33:15.884087 24562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:15.885037 24562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [term 1 LEADER]: Becoming Leader. State: Replica: 0e7f25dd5ba24a82b7b219d3a7cf2351, State: Running, Role: LEADER
I20260810 23:33:15.885612 24562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7f25dd5ba24a82b7b219d3a7cf2351" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 34445 } }
I20260810 23:33:15.886300 24556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:33:15.889055 24564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e7f25dd5ba24a82b7b219d3a7cf2351. Latest consensus state: current_term: 1 leader_uuid: "0e7f25dd5ba24a82b7b219d3a7cf2351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7f25dd5ba24a82b7b219d3a7cf2351" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 34445 } } }
I20260810 23:33:15.888967 24563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e7f25dd5ba24a82b7b219d3a7cf2351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e7f25dd5ba24a82b7b219d3a7cf2351" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 34445 } } }
I20260810 23:33:15.889422 24563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:15.889420 24564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351 [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:15.893563 24578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:33:15.898464 24578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:33:15.906710 24578 catalog_manager.cc:1347] Generated new cluster ID: 7476be7d5da34447861cf334dd6f92d2
I20260810 23:33:15.906836 24578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:33:15.924642 24578 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:33:15.925976 24578 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:33:15.940798 24578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e7f25dd5ba24a82b7b219d3a7cf2351: Generated new TSK 0
I20260810 23:33:15.941758 24578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:33:15.949930 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:34445
--builtin_ntp_servers=127.9.48.212:43125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 23:33:16.298431 24597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:16.298717 24597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:16.298873 24597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:16.306419 24597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:33:16.306515 24597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:16.306703 24597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:33:16.315850 24597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:43125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:34445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:33:16.317167 24597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:16.318969 24597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:16.331838 24602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:16.331897 24611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:16.331940 24606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:16.333294 24597 server_base.cc:1034] running on GCE node
I20260810 23:33:16.333892 24597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:16.334865 24597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:16.336074 24597 hybrid_clock.cc:648] HybridClock initialized: now 1786404796336020 us; error 40 us; skew 500 ppm
I20260810 23:33:16.336395 24597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:16.338600 24597 webserver.cc:460] Webserver started at http://127.9.48.193:36219/ using document root <none> and password file <none>
I20260810 23:33:16.339285 24597 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:16.339406 24597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:16.339754 24597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:16.342662 24597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "e57efd275efc43fbbb0311a2ccc5453f"
format_stamp: "Formatted at 2026-08-10 23:33:16 on dist-test-slave-04bb"
server_key: "4438d5d6b689a49e447e54792d633840"
server_key_iv: "a1ff2680f916636957f8873bdab56a5c"
server_key_version: "encryptionkey@0"
I20260810 23:33:16.343447 24597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "e57efd275efc43fbbb0311a2ccc5453f"
format_stamp: "Formatted at 2026-08-10 23:33:16 on dist-test-slave-04bb"
server_key: "4438d5d6b689a49e447e54792d633840"
server_key_iv: "a1ff2680f916636957f8873bdab56a5c"
server_key_version: "encryptionkey@0"
I20260810 23:33:16.348510 24597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 23:33:16.351892 24617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:16.353261 24597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:33:16.353415 24597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "e57efd275efc43fbbb0311a2ccc5453f"
format_stamp: "Formatted at 2026-08-10 23:33:16 on dist-test-slave-04bb"
server_key: "4438d5d6b689a49e447e54792d633840"
server_key_iv: "a1ff2680f916636957f8873bdab56a5c"
server_key_version: "encryptionkey@0"
I20260810 23:33:16.353586 24597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:16.382541 24597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:16.383358 24597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:16.383610 24597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:16.384593 24597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:33:16.386053 24597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:33:16.386143 24597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:16.386225 24597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:33:16.386327 24597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:16.421602 24597 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:40361
I20260810 23:33:16.421693 24803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:40361 every 8 connection(s)
I20260810 23:33:16.422991 24597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:33:16.430642 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 24597
I20260810 23:33:16.430843 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:33:16.431605 9411 external_mini_cluster.cc:1428] Setting key 6e12fffc9ca38eb46e547e530749126a
I20260810 23:33:16.434837 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:34445
--builtin_ntp_servers=127.9.48.212:43125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 23:33:16.436324 24809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:16.425686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:33133 (local address 127.9.48.254:34445)
0810 23:33:16.426130 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:16.426135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:16.427362 (+ 1227us) server_negotiation.cc:407] Connection header received
0810 23:33:16.429128 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:16.429132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:16.429209 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:16.429311 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:16.430421 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:16.431230 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:16.433330 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:16.433844 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:16.433938 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:16.435581 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:16.435585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:16.435588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:33:16.435606 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:16.435645 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:33:16.435657 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:16.435727 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:16.435991 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 23:33:16.436063 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":224,"threads_started":1}
I20260810 23:33:16.437795 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:16.426268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:34445 (local address 127.9.48.193:33133)
0810 23:33:16.427183 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:16.427219 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:16.428886 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:16.429514 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:16.429521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:16.429616 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:16.430240 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:16.430267 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:16.431393 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:16.431400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:16.433171 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:16.433185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:16.434070 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:16.434081 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:16.434391 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:16.435233 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:33:16.435239 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:16.435291 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:33:16.435382 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:33:16.435759 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:33:16.435773 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:33:16.435783 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 23:33:16.435972 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 23:33:16.436388 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":282,"threads_started":1}
I20260810 23:33:16.439293 24806 heartbeater.cc:347] Connected to a master server at 127.9.48.254:34445
I20260810 23:33:16.439625 24806 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:16.440462 24806 heartbeater.cc:511] Master 127.9.48.254:34445 requested a full tablet report, sending...
I20260810 23:33:16.442644 24497 ts_manager.cc:194] Registered new tserver with Master: e57efd275efc43fbbb0311a2ccc5453f (127.9.48.193:40361)
I20260810 23:33:16.444082 24497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.48.193:33133
W20260810 23:33:16.778012 24812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:16.778295 24812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:16.778381 24812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:16.785466 24812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:33:16.785559 24812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:16.785661 24812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:33:16.794756 24812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:43125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:34445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:33:16.795982 24812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:16.797758 24812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:16.810662 24819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:16.810699 24821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:16.810736 24823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:16.811515 24812 server_base.cc:1034] running on GCE node
I20260810 23:33:16.812158 24812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:16.813244 24812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:16.814450 24812 hybrid_clock.cc:648] HybridClock initialized: now 1786404796814406 us; error 31 us; skew 500 ppm
I20260810 23:33:16.814790 24812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:16.817262 24812 webserver.cc:460] Webserver started at http://127.9.48.194:37951/ using document root <none> and password file <none>
I20260810 23:33:16.818042 24812 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:16.818192 24812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:16.818547 24812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:16.821777 24812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "6bd928c6d70a45dea06b73ad30869b17"
format_stamp: "Formatted at 2026-08-10 23:33:16 on dist-test-slave-04bb"
server_key: "20738abf79fb65e59471f9bfe64fe598"
server_key_iv: "903f8e1cdb66ae3361c36a82dbad012f"
server_key_version: "encryptionkey@0"
I20260810 23:33:16.822645 24812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "6bd928c6d70a45dea06b73ad30869b17"
format_stamp: "Formatted at 2026-08-10 23:33:16 on dist-test-slave-04bb"
server_key: "20738abf79fb65e59471f9bfe64fe598"
server_key_iv: "903f8e1cdb66ae3361c36a82dbad012f"
server_key_version: "encryptionkey@0"
I20260810 23:33:16.828238 24812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:33:16.831954 24831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:16.833377 24812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:33:16.833585 24812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "6bd928c6d70a45dea06b73ad30869b17"
format_stamp: "Formatted at 2026-08-10 23:33:16 on dist-test-slave-04bb"
server_key: "20738abf79fb65e59471f9bfe64fe598"
server_key_iv: "903f8e1cdb66ae3361c36a82dbad012f"
server_key_version: "encryptionkey@0"
I20260810 23:33:16.833827 24812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:16.861562 24812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:16.862463 24812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:16.862798 24812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:16.863795 24812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:33:16.865439 24812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:33:16.865567 24812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:16.865696 24812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:33:16.865785 24812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:16.900493 24812 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:41979
I20260810 23:33:16.900629 25012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:41979 every 8 connection(s)
I20260810 23:33:16.901901 24812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:33:16.906000 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 24812
I20260810 23:33:16.906185 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:33:16.906673 9411 external_mini_cluster.cc:1428] Setting key 0a59a09553d14fcfbe5bd395cc65cfb2
I20260810 23:33:16.909535 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:34445
--builtin_ntp_servers=127.9.48.212:43125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 23:33:16.915876 24809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:16.904630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:45413 (local address 127.9.48.254:34445)
0810 23:33:16.904817 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:16.904822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:16.906307 (+ 1485us) server_negotiation.cc:407] Connection header received
0810 23:33:16.908374 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:16.908381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:16.908469 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:16.908638 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:16.909797 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:16.910491 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:16.913039 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:16.913440 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:16.913536 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:16.915154 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:16.915159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:16.915162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:33:16.915179 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:16.915210 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:33:16.915222 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:16.915293 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:16.915547 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 23:33:16.915624 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:33:16.917366 25015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:16.905143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:34445 (local address 127.9.48.194:45413)
0810 23:33:16.906073 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:16.906155 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:16.908097 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:16.908882 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:16.908890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:16.908981 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:16.909538 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:16.909554 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:16.910657 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:16.910666 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:16.912899 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:16.912918 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:16.913662 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:16.913671 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:16.914008 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:16.914795 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:33:16.914801 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:16.914854 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:33:16.914959 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:33:16.915322 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:33:16.915334 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:33:16.915344 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 23:33:16.915532 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 23:33:16.915951 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":328,"threads_started":1}
I20260810 23:33:16.918828 25013 heartbeater.cc:347] Connected to a master server at 127.9.48.254:34445
I20260810 23:33:16.919137 25013 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:16.919934 25013 heartbeater.cc:511] Master 127.9.48.254:34445 requested a full tablet report, sending...
I20260810 23:33:16.921492 24497 ts_manager.cc:194] Registered new tserver with Master: 6bd928c6d70a45dea06b73ad30869b17 (127.9.48.194:41979)
I20260810 23:33:16.922487 24497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.48.194:45413
W20260810 23:33:17.258303 25016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:17.258678 25016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:17.258847 25016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:17.265959 25016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:33:17.266083 25016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:17.266264 25016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:33:17.275187 25016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:43125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:34445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:33:17.276486 25016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:17.278316 25016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:17.291827 25024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:17.291971 25028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:17.292001 25025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:17.293156 25016 server_base.cc:1034] running on GCE node
I20260810 23:33:17.293690 25016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:17.294571 25016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:17.295869 25016 hybrid_clock.cc:648] HybridClock initialized: now 1786404797295803 us; error 59 us; skew 500 ppm
I20260810 23:33:17.296216 25016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:17.298477 25016 webserver.cc:460] Webserver started at http://127.9.48.195:45709/ using document root <none> and password file <none>
I20260810 23:33:17.299165 25016 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:17.299278 25016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:17.299643 25016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:17.302546 25016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "e16dd7c2fc3b4284a147cf9049a12346"
format_stamp: "Formatted at 2026-08-10 23:33:17 on dist-test-slave-04bb"
server_key: "8e5257deb76721b21d07dd1e0674dae5"
server_key_iv: "3d435fdba5981d2b96962998750eaeb7"
server_key_version: "encryptionkey@0"
I20260810 23:33:17.303396 25016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "e16dd7c2fc3b4284a147cf9049a12346"
format_stamp: "Formatted at 2026-08-10 23:33:17 on dist-test-slave-04bb"
server_key: "8e5257deb76721b21d07dd1e0674dae5"
server_key_iv: "3d435fdba5981d2b96962998750eaeb7"
server_key_version: "encryptionkey@0"
I20260810 23:33:17.308283 25016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260810 23:33:17.311579 25038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:17.312987 25016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:33:17.313162 25016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "e16dd7c2fc3b4284a147cf9049a12346"
format_stamp: "Formatted at 2026-08-10 23:33:17 on dist-test-slave-04bb"
server_key: "8e5257deb76721b21d07dd1e0674dae5"
server_key_iv: "3d435fdba5981d2b96962998750eaeb7"
server_key_version: "encryptionkey@0"
I20260810 23:33:17.313374 25016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:17.329540 25016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:17.330331 25016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:17.330627 25016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:17.331667 25016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:33:17.333297 25016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:33:17.333431 25016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:17.333564 25016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:33:17.333655 25016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:17.368443 25016 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:36159
I20260810 23:33:17.368544 25234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:36159 every 8 connection(s)
I20260810 23:33:17.369767 25016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:33:17.370353 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 25016
I20260810 23:33:17.370540 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:33:17.371176 9411 external_mini_cluster.cc:1428] Setting key a4787df49d4d0b98372df7342c5ef0cf
I20260810 23:33:17.382370 24809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:17.372620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:60205 (local address 127.9.48.254:34445)
0810 23:33:17.372806 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:17.372813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:17.374444 (+ 1631us) server_negotiation.cc:407] Connection header received
0810 23:33:17.376245 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:17.376250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:17.376317 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:17.376407 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:17.377496 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:17.378229 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:17.379685 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:17.380115 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:17.380213 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:17.381523 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:17.381528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:17.381531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:33:17.381549 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:17.381586 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:33:17.381610 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:17.381688 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:17.382025 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 23:33:17.382114 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:33:17.383386 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:17.373303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:34445 (local address 127.9.48.195:60205)
0810 23:33:17.374285 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:17.374322 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:17.375985 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:17.376676 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:17.376684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:17.376768 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:17.377336 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:17.377352 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:17.378352 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:17.378355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:17.379546 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:17.379558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:17.380300 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:17.380304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:17.380498 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:17.381216 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:33:17.381220 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:17.381255 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:33:17.381325 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:33:17.381747 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:33:17.381758 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:33:17.381766 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:33:17.381937 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 23:33:17.382255 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":344,"threads_started":1}
I20260810 23:33:17.384968 25237 heartbeater.cc:347] Connected to a master server at 127.9.48.254:34445
I20260810 23:33:17.385274 25237 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:17.386061 25237 heartbeater.cc:511] Master 127.9.48.254:34445 requested a full tablet report, sending...
I20260810 23:33:17.387650 24497 ts_manager.cc:194] Registered new tserver with Master: e16dd7c2fc3b4284a147cf9049a12346 (127.9.48.195:36159)
I20260810 23:33:17.388787 24497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.48.195:60205
I20260810 23:33:17.399011 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:33:17.405596 24809 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:17.404020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.9.48.254:34445)
0810 23:33:17.404170 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:17.404174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:17.404790 (+ 616us) server_negotiation.cc:407] Connection header received
0810 23:33:17.404909 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:17.404913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:17.404983 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:17.405062 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:17.405382 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44064: BlockingRecv error: recv got EOF from 127.0.0.1:44064 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:33:17.406214 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 24597
I20260810 23:33:17.414211 25250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:17.407660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37898 (local address 127.9.48.193:40361)
0810 23:33:17.408179 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:17.408186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:17.408212 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:33:17.408265 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:17.408270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:17.408453 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:17.408645 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:17.408959 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:17.409907 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:17.411117 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:17.411673 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:17.411777 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:17.412851 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:17.412860 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:17.412872 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:33:17.413542 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:17.413586 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:33:17.413605 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:17.413708 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:17.413807 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 23:33:17.413910 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":268,"threads_started":1}
I20260810 23:33:17.415498 24797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:33:17.496189 24806 heartbeater.cc:503] Master 127.9.48.254:34445 was elected leader, sending a full tablet report...
I20260810 23:33:17.501403 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 24597
W20260810 23:33:17.528482 24471 connection.cc:535] server connection from 127.9.48.193:33133 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:33:17.528703 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 24812
I20260810 23:33:17.536938 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:17.529681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.9.48.194:41979)
0810 23:33:17.530186 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:17.530193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:17.530223 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:33:17.530284 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:17.530290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:17.530506 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:17.530676 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:17.531144 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:17.532184 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:17.533583 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:17.534149 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:17.534298 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:17.535414 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:17.535423 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:17.535436 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:33:17.536041 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:17.536085 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:33:17.536106 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:17.536234 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:17.536418 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 23:33:17.536586 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":251,"threads_started":1}
I20260810 23:33:17.538380 25008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:33:17.620112 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 24812
I20260810 23:33:17.650331 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 25016
I20260810 23:33:17.658711 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:17.651317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.9.48.195:36159)
0810 23:33:17.652028 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:17.652035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:17.652068 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:33:17.652137 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:17.652143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:17.652378 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:17.652510 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:17.653046 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:17.654106 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:17.655348 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:17.655924 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:17.656021 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:17.657148 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:17.657156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:17.657170 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:33:17.657753 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:17.657802 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:33:17.657821 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:17.657915 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:17.658170 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 23:33:17.658322 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":392,"threads_started":1}
I20260810 23:33:17.660087 25228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:33:17.742542 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 25016
I20260810 23:33:17.768766 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 24447
I20260810 23:33:17.769985 24550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:33:17.890553 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 24447
2026-08-10T23:33:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2555 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T23:33:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:33:17Z Disabled control of system clock
I20260810 23:33:17.984004 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:45155
--webserver_interface=127.9.48.254
--webserver_port=0
--builtin_ntp_servers=127.9.48.212:44705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:45155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 23:33:18.322093 25266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:18.322403 25266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:18.322597 25266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:18.329689 25266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:33:18.329780 25266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:33:18.329887 25266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:18.329998 25266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:33:18.330067 25266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:33:18.338855 25266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:45155
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:45155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:33:18.340198 25266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:18.342123 25266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:18.354164 25273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:18.354120 25276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:18.354200 25274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:18.355063 25266 server_base.cc:1034] running on GCE node
I20260810 23:33:18.355831 25266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:18.357295 25266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:18.358541 25266 hybrid_clock.cc:648] HybridClock initialized: now 1786404798358507 us; error 43 us; skew 500 ppm
I20260810 23:33:18.358902 25266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:18.361194 25266 webserver.cc:460] Webserver started at http://127.9.48.254:45987/ using document root <none> and password file <none>
I20260810 23:33:18.361901 25266 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:18.362038 25266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:18.362382 25266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:18.365477 25266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "f2182726f60248819b64153d5d33f138"
format_stamp: "Formatted at 2026-08-10 23:33:18 on dist-test-slave-04bb"
server_key: "241019121984996f40bc11517531dfef"
server_key_iv: "fdf70773501de7259ac4dba30f87a061"
server_key_version: "encryptionkey@0"
I20260810 23:33:18.366331 25266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "f2182726f60248819b64153d5d33f138"
format_stamp: "Formatted at 2026-08-10 23:33:18 on dist-test-slave-04bb"
server_key: "241019121984996f40bc11517531dfef"
server_key_iv: "fdf70773501de7259ac4dba30f87a061"
server_key_version: "encryptionkey@0"
I20260810 23:33:18.371315 25266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:33:18.374752 25288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:18.376017 25266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 23:33:18.376196 25266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "f2182726f60248819b64153d5d33f138"
format_stamp: "Formatted at 2026-08-10 23:33:18 on dist-test-slave-04bb"
server_key: "241019121984996f40bc11517531dfef"
server_key_iv: "fdf70773501de7259ac4dba30f87a061"
server_key_version: "encryptionkey@0"
I20260810 23:33:18.376425 25266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:18.382015 25266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:18.382771 25266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:18.383080 25266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:18.402089 25266 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:45155
I20260810 23:33:18.402139 25370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:45155 every 8 connection(s)
I20260810 23:33:18.403405 25266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:33:18.407491 25371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:18.408253 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 25266
I20260810 23:33:18.408519 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:33:18.409502 9411 external_mini_cluster.cc:1428] Setting key 0e3a333833aeb3456a963b7b5f1bf5c5
I20260810 23:33:18.418144 25374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:18.411806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52698 (local address 127.9.48.254:45155)
0810 23:33:18.412900 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:18.412910 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:18.412955 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:33:18.414583 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:18.414605 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:18.415007 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:18.415414 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:18.416486 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:18.416492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:18.416505 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:33:18.416519 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:18.416640 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:33:18.416659 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:18.416897 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:18.417057 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 23:33:18.417299 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":387,"threads_started":1}
I20260810 23:33:18.422461 25371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138: Bootstrap starting.
I20260810 23:33:18.425613 25371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:18.427027 25371 log.cc:824] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138: Log is configured to *not* fsync() on all Append() calls
I20260810 23:33:18.430184 25371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138: No bootstrap required, opened a new log
I20260810 23:33:18.438062 25371 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2182726f60248819b64153d5d33f138" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45155 } }
I20260810 23:33:18.438462 25371 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:18.438542 25371 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2182726f60248819b64153d5d33f138, State: Initialized, Role: FOLLOWER
I20260810 23:33:18.439129 25371 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2182726f60248819b64153d5d33f138" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45155 } }
I20260810 23:33:18.439391 25371 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:33:18.439504 25371 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:33:18.439630 25371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:18.441973 25371 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2182726f60248819b64153d5d33f138" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45155 } }
I20260810 23:33:18.442461 25371 leader_election.cc:302] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2182726f60248819b64153d5d33f138; no voters:
I20260810 23:33:18.443027 25371 leader_election.cc:288] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:33:18.443181 25378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:18.443943 25378 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [term 1 LEADER]: Becoming Leader. State: Replica: f2182726f60248819b64153d5d33f138, State: Running, Role: LEADER
I20260810 23:33:18.444484 25378 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2182726f60248819b64153d5d33f138" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45155 } }
I20260810 23:33:18.445593 25371 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:33:18.447587 25380 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2182726f60248819b64153d5d33f138. Latest consensus state: current_term: 1 leader_uuid: "f2182726f60248819b64153d5d33f138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2182726f60248819b64153d5d33f138" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45155 } } }
I20260810 23:33:18.447979 25380 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:18.447710 25379 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2182726f60248819b64153d5d33f138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2182726f60248819b64153d5d33f138" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45155 } } }
I20260810 23:33:18.448186 25379 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138 [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:18.449748 25384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:33:18.455864 25384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:33:18.465241 25384 catalog_manager.cc:1347] Generated new cluster ID: 66299e0abeba49ef9fc6a234a2809857
I20260810 23:33:18.465356 25384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:33:18.484107 25384 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:33:18.484596 25384 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:33:18.499361 25384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2182726f60248819b64153d5d33f138: Generated new TSK 0
I20260810 23:33:18.500162 25384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:33:18.513005 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:45155
--builtin_ntp_servers=127.9.48.212:44705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 23:33:18.857858 25403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:18.858147 25403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:18.858235 25403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:18.865361 25403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:33:18.865463 25403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:18.865571 25403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:33:18.874514 25403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:45155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:33:18.875814 25403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:18.877609 25403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:18.891032 25411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:18.891151 25415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:18.891161 25412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:18.891734 25403 server_base.cc:1034] running on GCE node
I20260810 23:33:18.892357 25403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:18.893368 25403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:18.894621 25403 hybrid_clock.cc:648] HybridClock initialized: now 1786404798894553 us; error 68 us; skew 500 ppm
I20260810 23:33:18.894951 25403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:18.897338 25403 webserver.cc:460] Webserver started at http://127.9.48.193:41525/ using document root <none> and password file <none>
I20260810 23:33:18.898067 25403 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:18.898203 25403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:18.898550 25403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:18.901489 25403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "2438a24b55594bdb914a04a3be5901c9"
format_stamp: "Formatted at 2026-08-10 23:33:18 on dist-test-slave-04bb"
server_key: "2a1bfe354106c502800cbda8df7dc6d5"
server_key_iv: "76fd1527d7f5c92748a8f38f75d133ca"
server_key_version: "encryptionkey@0"
I20260810 23:33:18.902303 25403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "2438a24b55594bdb914a04a3be5901c9"
format_stamp: "Formatted at 2026-08-10 23:33:18 on dist-test-slave-04bb"
server_key: "2a1bfe354106c502800cbda8df7dc6d5"
server_key_iv: "76fd1527d7f5c92748a8f38f75d133ca"
server_key_version: "encryptionkey@0"
I20260810 23:33:18.907564 25403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:33:18.911011 25427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:18.912317 25403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:33:18.912492 25403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "2438a24b55594bdb914a04a3be5901c9"
format_stamp: "Formatted at 2026-08-10 23:33:18 on dist-test-slave-04bb"
server_key: "2a1bfe354106c502800cbda8df7dc6d5"
server_key_iv: "76fd1527d7f5c92748a8f38f75d133ca"
server_key_version: "encryptionkey@0"
I20260810 23:33:18.912794 25403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:18.918500 25403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:18.919268 25403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:18.919577 25403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:18.920481 25403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:33:18.921941 25403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:33:18.922065 25403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:18.922199 25403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:33:18.922293 25403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:18.956097 25403 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:34531
I20260810 23:33:18.956151 25612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:34531 every 8 connection(s)
I20260810 23:33:18.957499 25403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:33:18.966140 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 25403
I20260810 23:33:18.966403 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:33:18.967245 9411 external_mini_cluster.cc:1428] Setting key 0031d41f6b2cef28aa269782f557ecff
I20260810 23:33:18.967308 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:18.960726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:45155 (local address 127.9.48.193:47537)
0810 23:33:18.961638 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:18.961679 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:18.963254 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:18.964018 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:18.964026 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:18.964113 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:18.964906 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:33:18.964910 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:18.964962 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:33:18.965082 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:33:18.965606 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:33:18.965619 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:33:18.965628 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:33:18.965927 (+ 299us) client_negotiation.cc:240] Negotiation successful
0810 23:33:18.966179 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":280,"threads_started":1}
I20260810 23:33:18.968052 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:18.960223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:47537 (local address 127.9.48.254:45155)
0810 23:33:18.960715 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:18.960720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:18.961811 (+ 1091us) server_negotiation.cc:407] Connection header received
0810 23:33:18.963554 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:18.963559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:18.963649 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:18.963769 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:18.965285 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:18.965291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:18.965295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:33:18.965317 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:18.965391 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:33:18.965410 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:18.965542 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:18.967684 (+ 2142us) server_negotiation.cc:299] Negotiation successful
0810 23:33:18.967762 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":257,"threads_started":1}
I20260810 23:33:18.969359 25613 heartbeater.cc:347] Connected to a master server at 127.9.48.254:45155
I20260810 23:33:18.969671 25613 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:18.970268 25613 heartbeater.cc:511] Master 127.9.48.254:45155 requested a full tablet report, sending...
I20260810 23:33:18.970283 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:45155
--builtin_ntp_servers=127.9.48.212:44705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 23:33:18.972591 25312 ts_manager.cc:194] Registered new tserver with Master: 2438a24b55594bdb914a04a3be5901c9 (127.9.48.193:34531)
W20260810 23:33:19.307526 25618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:19.307876 25618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:19.308049 25618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:19.315325 25618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:33:19.315464 25618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:19.315649 25618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:33:19.324884 25618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:45155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:33:19.326190 25618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:19.328049 25618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:19.341224 25627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:19.341257 25633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:19.341508 25628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:19.342240 25618 server_base.cc:1034] running on GCE node
I20260810 23:33:19.342778 25618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:19.343675 25618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:19.344871 25618 hybrid_clock.cc:648] HybridClock initialized: now 1786404799344821 us; error 42 us; skew 500 ppm
I20260810 23:33:19.345202 25618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:19.351289 25618 webserver.cc:460] Webserver started at http://127.9.48.194:36133/ using document root <none> and password file <none>
I20260810 23:33:19.352085 25618 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:19.352178 25618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:19.352626 25618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:19.355536 25618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "c040ff40ff074a46be94b60986fbe683"
format_stamp: "Formatted at 2026-08-10 23:33:19 on dist-test-slave-04bb"
server_key: "db0dac7d88729f258ec6486f18f23f3e"
server_key_iv: "2eeb02b224831fe353db9c288d30ec46"
server_key_version: "encryptionkey@0"
I20260810 23:33:19.356388 25618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "c040ff40ff074a46be94b60986fbe683"
format_stamp: "Formatted at 2026-08-10 23:33:19 on dist-test-slave-04bb"
server_key: "db0dac7d88729f258ec6486f18f23f3e"
server_key_iv: "2eeb02b224831fe353db9c288d30ec46"
server_key_version: "encryptionkey@0"
I20260810 23:33:19.361624 25618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:33:19.365114 25647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:19.366448 25618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:33:19.366626 25618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "c040ff40ff074a46be94b60986fbe683"
format_stamp: "Formatted at 2026-08-10 23:33:19 on dist-test-slave-04bb"
server_key: "db0dac7d88729f258ec6486f18f23f3e"
server_key_iv: "2eeb02b224831fe353db9c288d30ec46"
server_key_version: "encryptionkey@0"
I20260810 23:33:19.366847 25618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:19.372593 25618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:19.373401 25618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:19.373683 25618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:19.374646 25618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:33:19.376107 25618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:33:19.376235 25618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:19.376380 25618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:33:19.376471 25618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:19.415086 25618 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:40595
I20260810 23:33:19.415095 25837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:40595 every 8 connection(s)
I20260810 23:33:19.416692 25618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:33:19.419305 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 25618
I20260810 23:33:19.419543 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:33:19.420092 9411 external_mini_cluster.cc:1428] Setting key f1278657a258b50fa4ec624532d81514
I20260810 23:33:19.423408 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:45155
--builtin_ntp_servers=127.9.48.212:44705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 23:33:19.426240 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:19.419850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:55195 (local address 127.9.48.254:45155)
0810 23:33:19.420055 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:19.420061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:19.421805 (+ 1744us) server_negotiation.cc:407] Connection header received
0810 23:33:19.423736 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:19.423741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:19.423833 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:19.423943 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:19.425439 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:19.425445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:19.425450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:33:19.425469 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:19.425511 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:33:19.425523 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:19.425648 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:19.425870 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 23:33:19.425940 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 23:33:19.427203 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:19.420439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:45155 (local address 127.9.48.194:55195)
0810 23:33:19.421638 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:19.421697 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:19.423448 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:19.424221 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:19.424233 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:19.424351 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:19.425135 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:33:19.425137 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:19.425176 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:33:19.425262 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:33:19.425631 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:33:19.425642 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:33:19.425650 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:33:19.425938 (+ 288us) client_negotiation.cc:240] Negotiation successful
0810 23:33:19.426194 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":350,"threads_started":1}
I20260810 23:33:19.428642 25838 heartbeater.cc:347] Connected to a master server at 127.9.48.254:45155
I20260810 23:33:19.428956 25838 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:19.429530 25838 heartbeater.cc:511] Master 127.9.48.254:45155 requested a full tablet report, sending...
I20260810 23:33:19.430917 25312 ts_manager.cc:194] Registered new tserver with Master: c040ff40ff074a46be94b60986fbe683 (127.9.48.194:40595)
W20260810 23:33:19.765273 25845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:19.765552 25845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:19.765726 25845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:19.773186 25845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:33:19.773293 25845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:19.773494 25845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:33:19.783221 25845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:45155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:33:19.784651 25845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:19.786514 25845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:19.800006 25861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:19.800180 25855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:19.800051 25857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:19.801229 25845 server_base.cc:1034] running on GCE node
I20260810 23:33:19.801847 25845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:19.802786 25845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:19.803990 25845 hybrid_clock.cc:648] HybridClock initialized: now 1786404799803935 us; error 41 us; skew 500 ppm
I20260810 23:33:19.804324 25845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:33:19.806746 25845 webserver.cc:460] Webserver started at http://127.9.48.195:35565/ using document root <none> and password file <none>
I20260810 23:33:19.807467 25845 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:19.807590 25845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:19.807961 25845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:19.810992 25845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "4d758fe8a5114b84b504970f0fd2615b"
format_stamp: "Formatted at 2026-08-10 23:33:19 on dist-test-slave-04bb"
server_key: "16c88dc5adc605a0621e3c4b4e0b00ac"
server_key_iv: "272e1427f1bbff7f14c537cb7b7d6284"
server_key_version: "encryptionkey@0"
I20260810 23:33:19.811798 25845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "4d758fe8a5114b84b504970f0fd2615b"
format_stamp: "Formatted at 2026-08-10 23:33:19 on dist-test-slave-04bb"
server_key: "16c88dc5adc605a0621e3c4b4e0b00ac"
server_key_iv: "272e1427f1bbff7f14c537cb7b7d6284"
server_key_version: "encryptionkey@0"
I20260810 23:33:19.817081 25845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260810 23:33:19.820709 25870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:19.822094 25845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:33:19.822319 25845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "4d758fe8a5114b84b504970f0fd2615b"
format_stamp: "Formatted at 2026-08-10 23:33:19 on dist-test-slave-04bb"
server_key: "16c88dc5adc605a0621e3c4b4e0b00ac"
server_key_iv: "272e1427f1bbff7f14c537cb7b7d6284"
server_key_version: "encryptionkey@0"
I20260810 23:33:19.822553 25845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:19.828368 25845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:19.829190 25845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:19.829473 25845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:19.830368 25845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:33:19.831740 25845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:33:19.831851 25845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:19.831959 25845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:33:19.832012 25845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:19.866542 25845 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:33783
I20260810 23:33:19.866601 26048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:33783 every 8 connection(s)
I20260810 23:33:19.867905 25845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:33:19.873036 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 25845
I20260810 23:33:19.873247 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:33:19.873807 9411 external_mini_cluster.cc:1428] Setting key 3ce2a7ef87ec2f8a4834166164212a86
I20260810 23:33:19.877250 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:19.870650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:37505 (local address 127.9.48.254:45155)
0810 23:33:19.870886 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:19.870893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:19.872630 (+ 1737us) server_negotiation.cc:407] Connection header received
0810 23:33:19.874964 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:19.874970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:19.875045 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:19.875162 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:19.876426 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:19.876432 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:19.876437 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:33:19.876456 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:19.876497 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:33:19.876508 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:19.876660 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:19.876882 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 23:33:19.876954 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:33:19.878198 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:19.871229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:45155 (local address 127.9.48.195:37505)
0810 23:33:19.872405 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:19.872458 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:19.874680 (+ 2222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:19.875407 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:19.875415 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:19.875509 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:19.876130 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:33:19.876132 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:19.876171 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:33:19.876256 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:33:19.876637 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:33:19.876648 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:33:19.876656 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:33:19.876970 (+ 314us) client_negotiation.cc:240] Negotiation successful
0810 23:33:19.877232 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":366,"threads_started":1}
I20260810 23:33:19.879529 26051 heartbeater.cc:347] Connected to a master server at 127.9.48.254:45155
I20260810 23:33:19.879856 26051 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:19.880568 26051 heartbeater.cc:511] Master 127.9.48.254:45155 requested a full tablet report, sending...
I20260810 23:33:19.882268 25312 ts_manager.cc:194] Registered new tserver with Master: 4d758fe8a5114b84b504970f0fd2615b (127.9.48.195:33783)
I20260810 23:33:19.889775 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:33:19.896111 25615 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:33:19.894464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.9.48.254:45155)
0810 23:33:19.894636 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:19.894641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:19.895215 (+ 574us) server_negotiation.cc:407] Connection header received
0810 23:33:19.895355 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:19.895359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:19.895412 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:19.895491 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:19.895876 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52704: BlockingRecv error: recv got EOF from 127.0.0.1:52704 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:33:19.897172 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 25403
I20260810 23:33:19.900736 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:19.898146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35590 (local address 127.9.48.193:34531)
0810 23:33:19.898689 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:19.898697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:19.898724 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:33:19.898782 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:19.898788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:19.898997 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:19.899130 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:19.900023 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:19.900030 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:19.900041 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:33:19.900054 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:19.900097 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:33:19.900116 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:19.900223 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:19.900294 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 23:33:19.900409 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":227,"threads_started":1}
I20260810 23:33:19.901952 25604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:33:19.983423 25613 heartbeater.cc:503] Master 127.9.48.254:45155 was elected leader, sending a full tablet report...
I20260810 23:33:19.984694 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 25403
W20260810 23:33:20.009771 25290 connection.cc:535] server connection from 127.9.48.193:47537 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:33:20.010146 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 25618
I20260810 23:33:20.013825 26069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:20.011183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.9.48.194:40595)
0810 23:33:20.011686 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:20.011694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:20.011722 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:33:20.011786 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:20.011792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:20.012013 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:20.012189 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:20.013115 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:20.013121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:20.013133 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:33:20.013145 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:20.013187 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:33:20.013205 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:20.013309 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:20.013410 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 23:33:20.013542 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":222,"threads_started":1}
I20260810 23:33:20.015039 25830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:33:20.098482 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 25618
I20260810 23:33:20.126561 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 25845
I20260810 23:33:20.130441 26071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:20.127536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.9.48.195:33783)
0810 23:33:20.128216 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:20.128225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:20.128256 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:33:20.128325 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:20.128332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:20.128620 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:20.128759 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:20.129808 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:20.129814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:20.129825 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:33:20.129837 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:20.129877 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:33:20.129895 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:20.130027 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:20.130079 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 23:33:20.130168 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":338,"threads_started":1}
I20260810 23:33:20.131682 26040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:33:20.213073 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 25845
I20260810 23:33:20.236513 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 25266
I20260810 23:33:20.237957 25364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:33:20.341264 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 25266
2026-08-10T23:33:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2453 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:33:20 dist-test-slave-04bb krb5kdc[26078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:33:20 dist-test-slave-04bb krb5kdc[26078](info): set up 2 sockets
Aug 10 23:33:20 dist-test-slave-04bb krb5kdc[26078](info): commencing operation
krb5kdc: starting...
W20260810 23:33:21.696192 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.302s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:33:21 dist-test-slave-04bb krb5kdc[26078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:33:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:33:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:33:21.979905 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:38469
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:38469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:33:22.321024 26099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:22.321338 26099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:22.321486 26099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:22.329088 26099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:33:22.329218 26099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:33:22.329304 26099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:22.329393 26099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:33:22.329456 26099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:33:22.339007 26099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:38469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:38469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:33:22.340304 26099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:22.341638 26099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:22.353861 26107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:22.353817 26117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:22.353897 26113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:22.355010 26099 server_base.cc:1034] running on GCE node
I20260810 23:33:22.355751 26099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:22.357200 26099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:22.358435 26099 hybrid_clock.cc:648] HybridClock initialized: now 1786404802358401 us; error 48 us; skew 500 ppm
I20260810 23:33:22.358784 26099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:33:22 dist-test-slave-04bb krb5kdc[26078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404802, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:33:22.362005 26099 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:33:22.363504 26099 webserver.cc:460] Webserver started at http://127.9.48.254:45101/ using document root <none> and password file <none>
I20260810 23:33:22.364195 26099 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:22.364344 26099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:22.364717 26099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:22.367697 26099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "71add4ecf6a04b7590e7300a1a8a3387"
format_stamp: "Formatted at 2026-08-10 23:33:22 on dist-test-slave-04bb"
server_key: "77b3212281a92792b7b9f3a8ed44b434"
server_key_iv: "e052e273f96e143c2ea2c6ea14d7d22e"
server_key_version: "encryptionkey@0"
I20260810 23:33:22.368487 26099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "71add4ecf6a04b7590e7300a1a8a3387"
format_stamp: "Formatted at 2026-08-10 23:33:22 on dist-test-slave-04bb"
server_key: "77b3212281a92792b7b9f3a8ed44b434"
server_key_iv: "e052e273f96e143c2ea2c6ea14d7d22e"
server_key_version: "encryptionkey@0"
I20260810 23:33:22.373585 26099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:33:22.377022 26130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:22.378278 26099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:33:22.378458 26099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "71add4ecf6a04b7590e7300a1a8a3387"
format_stamp: "Formatted at 2026-08-10 23:33:22 on dist-test-slave-04bb"
server_key: "77b3212281a92792b7b9f3a8ed44b434"
server_key_iv: "e052e273f96e143c2ea2c6ea14d7d22e"
server_key_version: "encryptionkey@0"
I20260810 23:33:22.378654 26099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:22.403256 26099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:22.407922 26099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:22.408362 26099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:22.427134 26099 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:38469
I20260810 23:33:22.427165 26203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:38469 every 8 connection(s)
I20260810 23:33:22.428625 26099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:33:22.429383 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 26099
I20260810 23:33:22.429638 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:33:22.430428 9411 external_mini_cluster.cc:1428] Setting key 5d990b08ab830db89d93d982c76e9e1e
I20260810 23:33:22.433246 26204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:33:22 dist-test-slave-04bb krb5kdc[26078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:33:22.446918 26204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387: Bootstrap starting.
I20260810 23:33:22.450809 26204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:22.452081 26204 log.cc:824] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387: Log is configured to *not* fsync() on all Append() calls
I20260810 23:33:22.454385 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:22.432417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.9.48.254:38469)
0810 23:33:22.433231 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:22.433246 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:22.433288 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:33:22.434954 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:22.434977 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:22.435340 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:22.435679 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:22.437680 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:22.439046 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:22.440980 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:22.441647 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:22.441772 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:22.446457 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:22.446468 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:22.446487 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:33:22.446541 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:22.449894 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:22.450734 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:22.450740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:22.450746 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:22.450970 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:22.451446 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:22.451449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:22.451451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:22.451975 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:22.452245 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:22.452827 (+ 582us) server_negotiation.cc:299] Negotiation successful
0810 23:33:22.453310 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":356,"threads_started":1}
I20260810 23:33:22.455287 26204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387: No bootstrap required, opened a new log
I20260810 23:33:22.463133 26204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71add4ecf6a04b7590e7300a1a8a3387" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38469 } }
I20260810 23:33:22.463480 26204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:22.463578 26204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71add4ecf6a04b7590e7300a1a8a3387, State: Initialized, Role: FOLLOWER
I20260810 23:33:22.464224 26204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71add4ecf6a04b7590e7300a1a8a3387" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38469 } }
I20260810 23:33:22.464476 26204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:33:22.464666 26204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:33:22.464850 26204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:22.467031 26204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71add4ecf6a04b7590e7300a1a8a3387" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38469 } }
I20260810 23:33:22.467540 26204 leader_election.cc:302] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71add4ecf6a04b7590e7300a1a8a3387; no voters:
I20260810 23:33:22.468220 26204 leader_election.cc:288] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:33:22.468393 26211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:22.469136 26211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [term 1 LEADER]: Becoming Leader. State: Replica: 71add4ecf6a04b7590e7300a1a8a3387, State: Running, Role: LEADER
I20260810 23:33:22.469662 26211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71add4ecf6a04b7590e7300a1a8a3387" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38469 } }
I20260810 23:33:22.470387 26204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:33:22.472702 26216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71add4ecf6a04b7590e7300a1a8a3387. Latest consensus state: current_term: 1 leader_uuid: "71add4ecf6a04b7590e7300a1a8a3387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71add4ecf6a04b7590e7300a1a8a3387" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38469 } } }
I20260810 23:33:22.472772 26214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71add4ecf6a04b7590e7300a1a8a3387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71add4ecf6a04b7590e7300a1a8a3387" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 38469 } } }
I20260810 23:33:22.473129 26214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:22.473129 26216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387 [sys.catalog]: This master's current role is: LEADER
I20260810 23:33:22.475297 26226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:33:22.479851 26226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:33:22.488812 26226 catalog_manager.cc:1347] Generated new cluster ID: 9a3e3c8c28354f9aa81e9fa1dc37cf20
I20260810 23:33:22.488920 26226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:33:22.514482 26226 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:33:22.515936 26226 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:33:22.523782 26226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387: Generated new TSK 0
I20260810 23:33:22.524653 26226 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:33:22.604343 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:38469
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:33:22.948021 26246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:22.948310 26246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:22.948508 26246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:22.955801 26246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:33:22.955917 26246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:22.956094 26246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:33:22.965626 26246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:38469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:33:22.966818 26246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:22.968104 26246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:22.981927 26257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:22.982049 26253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:22.981964 26254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:22.983685 26246 server_base.cc:1034] running on GCE node
I20260810 23:33:22.984273 26246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:22.985342 26246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:22.986547 26246 hybrid_clock.cc:648] HybridClock initialized: now 1786404802986498 us; error 43 us; skew 500 ppm
I20260810 23:33:22.986874 26246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:33:22 dist-test-slave-04bb krb5kdc[26078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404802, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:33:22.990315 26246 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:33:22.991756 26246 webserver.cc:460] Webserver started at http://127.9.48.193:40907/ using document root <none> and password file <none>
I20260810 23:33:22.992511 26246 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:22.992709 26246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:22.993067 26246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:22.996074 26246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "5e499197e9cc4c09badc59fb4882ce56"
format_stamp: "Formatted at 2026-08-10 23:33:22 on dist-test-slave-04bb"
server_key: "2168e76ad0aba1bf5e4bf9f6b51bc740"
server_key_iv: "d82e526baaaec3e1d970440daffac0bd"
server_key_version: "encryptionkey@0"
I20260810 23:33:22.996937 26246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "5e499197e9cc4c09badc59fb4882ce56"
format_stamp: "Formatted at 2026-08-10 23:33:22 on dist-test-slave-04bb"
server_key: "2168e76ad0aba1bf5e4bf9f6b51bc740"
server_key_iv: "d82e526baaaec3e1d970440daffac0bd"
server_key_version: "encryptionkey@0"
I20260810 23:33:23.002010 26246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 23:33:23.005575 26267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:23.006826 26246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:33:23.007004 26246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "5e499197e9cc4c09badc59fb4882ce56"
format_stamp: "Formatted at 2026-08-10 23:33:22 on dist-test-slave-04bb"
server_key: "2168e76ad0aba1bf5e4bf9f6b51bc740"
server_key_iv: "d82e526baaaec3e1d970440daffac0bd"
server_key_version: "encryptionkey@0"
I20260810 23:33:23.007210 26246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:23.023926 26246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:23.028349 26246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:23.028697 26246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:23.029623 26246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:33:23.030962 26246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:33:23.031068 26246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:23.031211 26246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:33:23.031281 26246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:23.063706 26246 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:46139
I20260810 23:33:23.063760 26444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:46139 every 8 connection(s)
I20260810 23:33:23.065018 26246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:33:23.073966 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 26246
I20260810 23:33:23.074206 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:33:23.075028 9411 external_mini_cluster.cc:1428] Setting key 0b42cd40fa818b957461d3dc9f31ed6a
Aug 10 23:33:23 dist-test-slave-04bb krb5kdc[26078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404802, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:33:23.086517 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:23.067450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:56689 (local address 127.9.48.254:38469)
0810 23:33:23.067921 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:23.067926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:23.068722 (+ 796us) server_negotiation.cc:407] Connection header received
0810 23:33:23.070575 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:23.070581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:23.070673 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:23.070785 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:23.072370 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:23.073122 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:23.075468 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:23.076039 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:23.076147 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:23.080263 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:23.080268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:23.080271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:33:23.080304 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:23.083232 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:23.084168 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:23.084172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:23.084175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:23.084296 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:23.084822 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:23.084826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:23.084828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:23.085080 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:23.085194 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:23.086091 (+ 897us) server_negotiation.cc:299] Negotiation successful
0810 23:33:23.086255 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":254,"threads_started":1}
I20260810 23:33:23.088094 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:23.067835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:38469 (local address 127.9.48.193:56689)
0810 23:33:23.068559 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:23.068597 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:23.070291 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:23.071030 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:23.071038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:23.071623 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:23.072184 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:23.072201 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:23.073281 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:23.073285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:23.075314 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:23.075330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:23.076324 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:23.076331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:23.076582 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:23.077250 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:23.077273 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:23.080034 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:33:23.083377 (+ 3343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:33:23.083389 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:33:23.083406 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:33:23.084059 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:33:23.084408 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:33:23.084416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:33:23.084421 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:33:23.084708 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:33:23.085215 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:33:23.085225 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:33:23.085684 (+ 459us) client_negotiation.cc:769] Sending connection context
0810 23:33:23.086004 (+ 320us) client_negotiation.cc:240] Negotiation successful
0810 23:33:23.086604 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":273,"threads_started":1}
I20260810 23:33:23.089768 26445 heartbeater.cc:347] Connected to a master server at 127.9.48.254:38469
I20260810 23:33:23.090090 26445 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:23.090978 26445 heartbeater.cc:511] Master 127.9.48.254:38469 requested a full tablet report, sending...
I20260810 23:33:23.093318 26160 ts_manager.cc:194] Registered new tserver with Master: 5e499197e9cc4c09badc59fb4882ce56 (127.9.48.193:46139)
I20260810 23:33:23.094794 26160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:56689
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:33:23.141058 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:38469
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:33:23.486121 26452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:23.486429 26452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:23.486577 26452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:23.493646 26452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:33:23.493805 26452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:23.493997 26452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:33:23.503391 26452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:38469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:33:23.504671 26452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:23.506095 26452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:23.519042 26465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:23.519335 26459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:23.519057 26467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:23.520114 26452 server_base.cc:1034] running on GCE node
I20260810 23:33:23.520767 26452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:23.521589 26452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:23.522840 26452 hybrid_clock.cc:648] HybridClock initialized: now 1786404803522757 us; error 65 us; skew 500 ppm
I20260810 23:33:23.523166 26452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:33:23 dist-test-slave-04bb krb5kdc[26078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404803, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:33:23.526489 26452 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:33:23.527953 26452 webserver.cc:460] Webserver started at http://127.9.48.194:37457/ using document root <none> and password file <none>
I20260810 23:33:23.528687 26452 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:23.528825 26452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:23.529191 26452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:23.532110 26452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "b821f891ad1a40d98f7ba22b338a8832"
format_stamp: "Formatted at 2026-08-10 23:33:23 on dist-test-slave-04bb"
server_key: "ebc08875519eae9a5c8be88617d08267"
server_key_iv: "ed6e35c81a2c39c7ca650542a380e9d7"
server_key_version: "encryptionkey@0"
I20260810 23:33:23.532929 26452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "b821f891ad1a40d98f7ba22b338a8832"
format_stamp: "Formatted at 2026-08-10 23:33:23 on dist-test-slave-04bb"
server_key: "ebc08875519eae9a5c8be88617d08267"
server_key_iv: "ed6e35c81a2c39c7ca650542a380e9d7"
server_key_version: "encryptionkey@0"
I20260810 23:33:23.537926 26452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 23:33:23.541355 26475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:23.542693 26452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:33:23.542886 26452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "b821f891ad1a40d98f7ba22b338a8832"
format_stamp: "Formatted at 2026-08-10 23:33:23 on dist-test-slave-04bb"
server_key: "ebc08875519eae9a5c8be88617d08267"
server_key_iv: "ed6e35c81a2c39c7ca650542a380e9d7"
server_key_version: "encryptionkey@0"
I20260810 23:33:23.543078 26452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:23.563655 26452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:23.568001 26452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:23.568287 26452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:23.569319 26452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:33:23.570731 26452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:33:23.570875 26452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:23.571004 26452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:33:23.571055 26452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:23.605788 26452 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:46173
I20260810 23:33:23.605875 26646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:46173 every 8 connection(s)
I20260810 23:33:23.607144 26452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:33:23.608701 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 26452
I20260810 23:33:23.608919 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:33:23.609467 9411 external_mini_cluster.cc:1428] Setting key c1eaa25f7bb484b076a1c2ac3dfaa84d
Aug 10 23:33:23 dist-test-slave-04bb krb5kdc[26078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404803, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:33:23.628785 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:23.609963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:39933 (local address 127.9.48.254:38469)
0810 23:33:23.610551 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:23.610561 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:23.612590 (+ 2029us) server_negotiation.cc:407] Connection header received
0810 23:33:23.613630 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:23.613635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:23.613728 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:23.613844 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:23.615308 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:23.616025 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:23.617667 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:23.618432 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:23.618585 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:23.622010 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:23.622019 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:23.622027 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:33:23.622073 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:23.625236 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:23.625971 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:23.625979 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:23.625985 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:23.626152 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:23.626664 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:23.626672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:23.626678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:23.627059 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:23.627217 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:23.627959 (+ 742us) server_negotiation.cc:299] Negotiation successful
0810 23:33:23.628214 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":297,"threads_started":1}
I20260810 23:33:23.629380 26656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:23.610628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:38469 (local address 127.9.48.194:39933)
0810 23:33:23.611460 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:23.611529 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:23.613373 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:23.614083 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:23.614090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:23.614577 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:23.615139 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:23.615157 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:23.616190 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:23.616195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:23.617487 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:23.617506 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:23.618658 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:23.618663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:23.618860 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:23.619579 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:23.619606 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:23.621779 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:33:23.625387 (+ 3608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:33:23.625396 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:33:23.625411 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:33:23.625838 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:33:23.626271 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:33:23.626275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:33:23.626278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:33:23.626518 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:33:23.627218 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:33:23.627224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:33:23.627538 (+ 314us) client_negotiation.cc:769] Sending connection context
0810 23:33:23.627803 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 23:33:23.628208 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":345,"threads_started":1}
I20260810 23:33:23.630951 26647 heartbeater.cc:347] Connected to a master server at 127.9.48.254:38469
I20260810 23:33:23.631276 26647 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:23.632249 26647 heartbeater.cc:511] Master 127.9.48.254:38469 requested a full tablet report, sending...
I20260810 23:33:23.634007 26160 ts_manager.cc:194] Registered new tserver with Master: b821f891ad1a40d98f7ba22b338a8832 (127.9.48.194:46173)
I20260810 23:33:23.635020 26160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:39933
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:33:23.680119 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:38469
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:33:24.023059 26662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:33:24.023386 26662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:33:24.023553 26662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:33:24.030786 26662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:33:24.030938 26662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:33:24.031122 26662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:33:24.040990 26662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:38469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:33:24.042325 26662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:33:24.043661 26662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:33:24.057014 26673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:24.057014 26674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:33:24.057292 26679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:33:24.058411 26662 server_base.cc:1034] running on GCE node
I20260810 23:33:24.059027 26662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:33:24.059966 26662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:33:24.061187 26662 hybrid_clock.cc:648] HybridClock initialized: now 1786404804061126 us; error 73 us; skew 500 ppm
I20260810 23:33:24.061497 26662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:33:24 dist-test-slave-04bb krb5kdc[26078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404804, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:33:24.064729 26662 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:33:24.066411 26662 webserver.cc:460] Webserver started at http://127.9.48.195:36783/ using document root <none> and password file <none>
I20260810 23:33:24.067095 26662 fs_manager.cc:359] Metadata directory not provided
I20260810 23:33:24.067245 26662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:33:24.067550 26662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:33:24.070552 26662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079"
format_stamp: "Formatted at 2026-08-10 23:33:24 on dist-test-slave-04bb"
server_key: "d2bf166072f9962f8914934155be4b04"
server_key_iv: "afeb9d5468d84e3380c5d9dcd5cdaa4d"
server_key_version: "encryptionkey@0"
I20260810 23:33:24.071401 26662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079"
format_stamp: "Formatted at 2026-08-10 23:33:24 on dist-test-slave-04bb"
server_key: "d2bf166072f9962f8914934155be4b04"
server_key_iv: "afeb9d5468d84e3380c5d9dcd5cdaa4d"
server_key_version: "encryptionkey@0"
I20260810 23:33:24.076484 26662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:33:24.079993 26692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:24.081460 26662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:33:24.081635 26662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079"
format_stamp: "Formatted at 2026-08-10 23:33:24 on dist-test-slave-04bb"
server_key: "d2bf166072f9962f8914934155be4b04"
server_key_iv: "afeb9d5468d84e3380c5d9dcd5cdaa4d"
server_key_version: "encryptionkey@0"
I20260810 23:33:24.081846 26662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:33:24.098407 26445 heartbeater.cc:503] Master 127.9.48.254:38469 was elected leader, sending a full tablet report...
I20260810 23:33:24.103029 26662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:33:24.107029 26662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:33:24.107378 26662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:33:24.108338 26662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:33:24.109825 26662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:33:24.109928 26662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:24.110069 26662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:33:24.110138 26662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:33:24.145691 26662 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:46493
I20260810 23:33:24.145735 26866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:46493 every 8 connection(s)
I20260810 23:33:24.147007 26662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:33:24.148276 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 26662
I20260810 23:33:24.148496 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:33:24.149000 9411 external_mini_cluster.cc:1428] Setting key f8953c4a58d3bc05a33eb96b7f94612e
Aug 10 23:33:24 dist-test-slave-04bb krb5kdc[26078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404804, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:33:24.167363 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.149984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:38277 (local address 127.9.48.254:38469)
0810 23:33:24.150794 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:24.150801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:24.151629 (+ 828us) server_negotiation.cc:407] Connection header received
0810 23:33:24.153555 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:24.153559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:24.153636 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:24.153729 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:24.155249 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.156047 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.157655 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.158032 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.158127 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.161501 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:24.161506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:24.161510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:33:24.161550 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:24.164621 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:24.165415 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:24.165419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:24.165421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:24.165547 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:24.165959 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:24.165962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:24.165964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:24.166215 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:24.166329 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:24.166905 (+ 576us) server_negotiation.cc:299] Negotiation successful
0810 23:33:24.167092 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":554,"threads_started":1}
I20260810 23:33:24.168311 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.150550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:38469 (local address 127.9.48.195:38277)
0810 23:33:24.151463 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:24.151512 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:24.153310 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:24.153998 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:24.154006 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:24.154530 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:33:24.155096 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:24.155114 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.156197 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.156201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:24.157531 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:24.157544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.158292 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.158299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:24.158500 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.159121 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:33:24.159148 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:33:24.161302 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:33:24.164753 (+ 3451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:33:24.164763 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:33:24.164777 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:33:24.165296 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:33:24.165639 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:33:24.165642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:33:24.165644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:33:24.165861 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:33:24.166317 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:33:24.166322 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:33:24.166602 (+ 280us) client_negotiation.cc:769] Sending connection context
0810 23:33:24.166819 (+ 217us) client_negotiation.cc:240] Negotiation successful
0810 23:33:24.167201 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":373,"threads_started":1}
I20260810 23:33:24.169691 26867 heartbeater.cc:347] Connected to a master server at 127.9.48.254:38469
I20260810 23:33:24.169952 26867 heartbeater.cc:464] Registering TS with master...
I20260810 23:33:24.170689 26867 heartbeater.cc:511] Master 127.9.48.254:38469 requested a full tablet report, sending...
I20260810 23:33:24.172097 26160 ts_manager.cc:194] Registered new tserver with Master: d2bfc1f7bd60469d90b2db6b5a8dd079 (127.9.48.195:46493)
I20260810 23:33:24.172987 26160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:38277
I20260810 23:33:24.177456 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:33:24.195240 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.182536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.9.48.254:38469)
0810 23:33:24.182705 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:24.182710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:24.183283 (+ 573us) server_negotiation.cc:407] Connection header received
0810 23:33:24.183499 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:24.183504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:24.183565 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:24.183665 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:24.185195 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.186012 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.187310 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.187754 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.187900 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.190205 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:24.190216 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:24.190219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:33:24.190249 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:24.192580 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:24.193304 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:24.193308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:24.193311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:24.193437 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:24.193805 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:24.193811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:24.193814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:24.194118 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:24.194394 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:24.194727 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 23:33:24.194955 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:33:24.199095 26160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:33:24.205981 26160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:33:24.239845 26894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.218441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.9.48.193:46139)
0810 23:33:24.230089 (+ 11648us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:24.230101 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:24.230132 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:33:24.230250 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:24.230257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:24.230563 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:24.231805 (+ 1242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:33:24.232144 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.234240 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.237377 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.238518 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.238656 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.238899 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:24.239307 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 23:33:24.239490 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11459,"thread_start_us":12046,"threads_started":1}
I20260810 23:33:24.239899 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.218105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46139 (local address 127.0.0.1:52892)
0810 23:33:24.229377 (+ 11272us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:24.229394 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:24.229486 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:24.230867 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:24.230879 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:24.230943 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:33:24.231519 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:24.231536 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.235035 (+ 3499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.235040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:24.237227 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:24.237239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.238772 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.238780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:24.238995 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.239088 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 23:33:24.239252 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 23:33:24.239449 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11156,"thread_start_us":10185,"threads_started":1}
I20260810 23:33:24.243429 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.230251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.9.48.194:46173)
0810 23:33:24.234237 (+ 3986us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:24.234250 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:24.234780 (+ 530us) server_negotiation.cc:407] Connection header received
0810 23:33:24.234906 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:24.234914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:24.235151 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:24.235374 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:33:24.236473 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.238059 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.240932 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.242379 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.242539 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.242725 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:24.242986 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 23:33:24.243137 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3770,"thread_start_us":3641,"threads_started":1}
I20260810 23:33:24.243472 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.230922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:46173 (local address 127.0.0.1:34892)
0810 23:33:24.234508 (+ 3586us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:24.234529 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:24.234662 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:24.235583 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:24.235587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:24.235612 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:33:24.236273 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:24.236287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.238267 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.238271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:24.240765 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:24.240781 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.242610 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.242618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:24.242800 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.242834 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:33:24.242924 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 23:33:24.243049 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3450,"mutex_wait_us":88,"thread_start_us":3345,"threads_started":1}
I20260810 23:33:24.244012 26893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.217906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:46493 (local address 127.0.0.1:55964)
0810 23:33:24.229200 (+ 11294us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:24.229248 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:24.229438 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:24.235532 (+ 6094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:24.235536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:24.235564 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:33:24.236144 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:24.236155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.238203 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.238211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:24.241536 (+ 3325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:24.241548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.243194 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.243200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:24.243378 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.243419 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 23:33:24.243519 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 23:33:24.243657 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10529,"mutex_wait_us":157,"thread_start_us":11672,"threads_started":1}
I20260810 23:33:24.244341 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.218471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.9.48.195:46493)
0810 23:33:24.234897 (+ 16426us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:24.234905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:24.234940 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:33:24.235036 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:24.235043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:24.235301 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:24.235554 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:33:24.236330 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.238052 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.241685 (+ 3633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.242695 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.242815 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.243037 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:24.243842 (+ 805us) server_negotiation.cc:299] Negotiation successful
0810 23:33:24.244012 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16209,"thread_start_us":16031,"threads_started":1}
I20260810 23:33:24.244379 26347 tablet_service.cc:1467] Processing CreateTablet for tablet 7d51932cd8b741e391af0c0a2a1138a3 (DEFAULT_TABLE table=test-table [id=b42385a1b1814baaa1a2bcca6c99391b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:24.245942 26347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d51932cd8b741e391af0c0a2a1138a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:24.246992 26544 tablet_service.cc:1467] Processing CreateTablet for tablet 7d51932cd8b741e391af0c0a2a1138a3 (DEFAULT_TABLE table=test-table [id=b42385a1b1814baaa1a2bcca6c99391b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:24.248281 26544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d51932cd8b741e391af0c0a2a1138a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:24.248015 26759 tablet_service.cc:1467] Processing CreateTablet for tablet 7d51932cd8b741e391af0c0a2a1138a3 (DEFAULT_TABLE table=test-table [id=b42385a1b1814baaa1a2bcca6c99391b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:33:24.249423 26759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d51932cd8b741e391af0c0a2a1138a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:33:24.259835 26900 tablet_bootstrap.cc:492] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832: Bootstrap starting.
I20260810 23:33:24.262833 26900 tablet_bootstrap.cc:654] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:24.263931 26900 log.cc:824] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832: Log is configured to *not* fsync() on all Append() calls
I20260810 23:33:24.264609 26902 tablet_bootstrap.cc:492] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079: Bootstrap starting.
I20260810 23:33:24.268544 26902 tablet_bootstrap.cc:654] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:24.268625 26901 tablet_bootstrap.cc:492] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56: Bootstrap starting.
I20260810 23:33:24.268698 26900 tablet_bootstrap.cc:492] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832: No bootstrap required, opened a new log
I20260810 23:33:24.269029 26900 ts_tablet_manager.cc:1397] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260810 23:33:24.269809 26902 log.cc:824] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079: Log is configured to *not* fsync() on all Append() calls
I20260810 23:33:24.271507 26901 tablet_bootstrap.cc:654] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56: Neither blocks nor log segments found. Creating new log.
I20260810 23:33:24.272473 26901 log.cc:824] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56: Log is configured to *not* fsync() on all Append() calls
I20260810 23:33:24.278726 26901 tablet_bootstrap.cc:492] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56: No bootstrap required, opened a new log
I20260810 23:33:24.278733 26902 tablet_bootstrap.cc:492] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079: No bootstrap required, opened a new log
I20260810 23:33:24.279000 26901 ts_tablet_manager.cc:1397] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260810 23:33:24.279045 26902 ts_tablet_manager.cc:1397] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.010s
I20260810 23:33:24.280022 26900 raft_consensus.cc:348] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46493 } } peers { permanent_uuid: "5e499197e9cc4c09badc59fb4882ce56" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46139 } } peers { permanent_uuid: "b821f891ad1a40d98f7ba22b338a8832" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46173 } }
I20260810 23:33:24.280527 26900 raft_consensus.cc:374] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:24.280718 26900 raft_consensus.cc:729] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b821f891ad1a40d98f7ba22b338a8832, State: Initialized, Role: FOLLOWER
I20260810 23:33:24.281467 26900 consensus_queue.cc:260] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46493 } } peers { permanent_uuid: "5e499197e9cc4c09badc59fb4882ce56" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46139 } } peers { permanent_uuid: "b821f891ad1a40d98f7ba22b338a8832" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46173 } }
I20260810 23:33:24.282677 26647 heartbeater.cc:503] Master 127.9.48.254:38469 was elected leader, sending a full tablet report...
I20260810 23:33:24.283411 26900 ts_tablet_manager.cc:1428] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260810 23:33:24.289952 26901 raft_consensus.cc:348] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46493 } } peers { permanent_uuid: "5e499197e9cc4c09badc59fb4882ce56" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46139 } } peers { permanent_uuid: "b821f891ad1a40d98f7ba22b338a8832" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46173 } }
I20260810 23:33:24.290418 26901 raft_consensus.cc:374] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:24.290522 26901 raft_consensus.cc:729] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e499197e9cc4c09badc59fb4882ce56, State: Initialized, Role: FOLLOWER
I20260810 23:33:24.290493 26902 raft_consensus.cc:348] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46493 } } peers { permanent_uuid: "5e499197e9cc4c09badc59fb4882ce56" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46139 } } peers { permanent_uuid: "b821f891ad1a40d98f7ba22b338a8832" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46173 } }
I20260810 23:33:24.290972 26902 raft_consensus.cc:374] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:33:24.291082 26902 raft_consensus.cc:729] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bfc1f7bd60469d90b2db6b5a8dd079, State: Initialized, Role: FOLLOWER
I20260810 23:33:24.291191 26901 consensus_queue.cc:260] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46493 } } peers { permanent_uuid: "5e499197e9cc4c09badc59fb4882ce56" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46139 } } peers { permanent_uuid: "b821f891ad1a40d98f7ba22b338a8832" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46173 } }
I20260810 23:33:24.291702 26902 consensus_queue.cc:260] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46493 } } peers { permanent_uuid: "5e499197e9cc4c09badc59fb4882ce56" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46139 } } peers { permanent_uuid: "b821f891ad1a40d98f7ba22b338a8832" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46173 } }
I20260810 23:33:24.293298 26901 ts_tablet_manager.cc:1428] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56: Time spent starting tablet: real 0.014s user 0.010s sys 0.006s
I20260810 23:33:24.293857 26867 heartbeater.cc:503] Master 127.9.48.254:38469 was elected leader, sending a full tablet report...
I20260810 23:33:24.294492 26902 ts_tablet_manager.cc:1428] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
W20260810 23:33:24.317518 26446 tablet.cc:2369] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:33:24.359310 26650 tablet.cc:2369] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:33:24.398977 26868 tablet.cc:2369] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:33:24.552461 26911 raft_consensus.cc:482] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:33:24.552901 26911 raft_consensus.cc:504] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46493 } } peers { permanent_uuid: "5e499197e9cc4c09badc59fb4882ce56" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46139 } } peers { permanent_uuid: "b821f891ad1a40d98f7ba22b338a8832" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46173 } }
I20260810 23:33:24.554675 26911 leader_election.cc:288] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2bfc1f7bd60469d90b2db6b5a8dd079 (127.9.48.195:46493), 5e499197e9cc4c09badc59fb4882ce56 (127.9.48.193:46139)
I20260810 23:33:24.561717 26894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.555320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:56479 (local address 127.9.48.193:46139)
0810 23:33:24.555556 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:24.555563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:24.556162 (+ 599us) server_negotiation.cc:407] Connection header received
0810 23:33:24.556339 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:24.556345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:24.556422 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:24.556518 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:33:24.557171 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.557968 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.559852 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.560725 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.560850 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.560937 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:24.561310 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 23:33:24.561440 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 23:33:24.561722 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.555160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:54451 (local address 127.9.48.195:46493)
0810 23:33:24.555401 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:24.555410 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:24.556356 (+ 946us) server_negotiation.cc:407] Connection header received
0810 23:33:24.556563 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:24.556569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:24.556653 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:24.556749 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:33:24.557432 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.558214 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.559951 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.560816 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.560980 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.561053 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:24.561364 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 23:33:24.561495 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:33:24.561761 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.555181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46139 (local address 127.9.48.194:56479)
0810 23:33:24.556046 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:24.556062 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:24.556187 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:24.556565 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:24.556569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:24.556597 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:33:24.557006 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:24.557018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.558104 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.558109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:24.559672 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:24.559685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.560957 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.560965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:24.561140 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.561165 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:33:24.561243 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 23:33:24.561377 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":512,"threads_started":1}
I20260810 23:33:24.561884 26915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.555241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:46493 (local address 127.9.48.194:54451)
0810 23:33:24.556213 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0810 23:33:24.556250 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:33:24.556397 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:33:24.556793 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:33:24.556797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:33:24.556817 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:33:24.557272 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:24.557284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.558355 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.558359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:24.559791 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:33:24.559808 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.561039 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.561043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:33:24.561178 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.561200 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:33:24.561315 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 23:33:24.561438 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":2,"thread_start_us":535,"threads_started":1}
I20260810 23:33:24.562835 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d51932cd8b741e391af0c0a2a1138a3" candidate_uuid: "b821f891ad1a40d98f7ba22b338a8832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e499197e9cc4c09badc59fb4882ce56" is_pre_election: true
I20260810 23:33:24.562845 26795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d51932cd8b741e391af0c0a2a1138a3" candidate_uuid: "b821f891ad1a40d98f7ba22b338a8832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079" is_pre_election: true
I20260810 23:33:24.563339 26379 raft_consensus.cc:2454] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b821f891ad1a40d98f7ba22b338a8832 in term 0.
I20260810 23:33:24.563354 26795 raft_consensus.cc:2454] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b821f891ad1a40d98f7ba22b338a8832 in term 0.
I20260810 23:33:24.564013 26478 leader_election.cc:302] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b821f891ad1a40d98f7ba22b338a8832, d2bfc1f7bd60469d90b2db6b5a8dd079; no voters:
I20260810 23:33:24.564411 26911 raft_consensus.cc:2781] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:33:24.564565 26911 raft_consensus.cc:482] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:33:24.564685 26911 raft_consensus.cc:3037] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:24.566900 26911 raft_consensus.cc:504] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46493 } } peers { permanent_uuid: "5e499197e9cc4c09badc59fb4882ce56" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46139 } } peers { permanent_uuid: "b821f891ad1a40d98f7ba22b338a8832" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46173 } }
I20260810 23:33:24.567483 26911 leader_election.cc:288] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [CANDIDATE]: Term 1 election: Requested vote from peers d2bfc1f7bd60469d90b2db6b5a8dd079 (127.9.48.195:46493), 5e499197e9cc4c09badc59fb4882ce56 (127.9.48.193:46139)
I20260810 23:33:24.567983 26379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d51932cd8b741e391af0c0a2a1138a3" candidate_uuid: "b821f891ad1a40d98f7ba22b338a8832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e499197e9cc4c09badc59fb4882ce56"
I20260810 23:33:24.567983 26795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d51932cd8b741e391af0c0a2a1138a3" candidate_uuid: "b821f891ad1a40d98f7ba22b338a8832" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079"
I20260810 23:33:24.568209 26795 raft_consensus.cc:3037] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:24.568205 26379 raft_consensus.cc:3037] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:33:24.571523 26795 raft_consensus.cc:2454] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b821f891ad1a40d98f7ba22b338a8832 in term 1.
I20260810 23:33:24.571523 26379 raft_consensus.cc:2454] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b821f891ad1a40d98f7ba22b338a8832 in term 1.
I20260810 23:33:24.571974 26481 leader_election.cc:302] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e499197e9cc4c09badc59fb4882ce56, b821f891ad1a40d98f7ba22b338a8832; no voters:
I20260810 23:33:24.572273 26911 raft_consensus.cc:2781] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:33:24.572773 26911 raft_consensus.cc:686] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [term 1 LEADER]: Becoming Leader. State: Replica: b821f891ad1a40d98f7ba22b338a8832, State: Running, Role: LEADER
I20260810 23:33:24.573249 26911 consensus_queue.cc:237] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46493 } } peers { permanent_uuid: "5e499197e9cc4c09badc59fb4882ce56" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46139 } } peers { permanent_uuid: "b821f891ad1a40d98f7ba22b338a8832" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46173 } }
I20260810 23:33:24.578142 26159 catalog_manager.cc:5495] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 reported cstate change: term changed from 0 to 1, leader changed from <none> to b821f891ad1a40d98f7ba22b338a8832 (127.9.48.194). New cstate: current_term: 1 leader_uuid: "b821f891ad1a40d98f7ba22b338a8832" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e499197e9cc4c09badc59fb4882ce56" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b821f891ad1a40d98f7ba22b338a8832" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46173 } health_report { overall_health: HEALTHY } } }
I20260810 23:33:24.683198 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.677684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34900 (local address 127.9.48.194:46173)
0810 23:33:24.677906 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:24.677911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:24.678008 (+ 97us) server_negotiation.cc:407] Connection header received
0810 23:33:24.678228 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:24.678232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:24.678329 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:24.678460 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:33:24.678859 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.679612 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.681190 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.681585 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.681675 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.682145 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:33:24.682638 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:33:24.682749 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:24.682916 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 23:33:24.682982 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:33:24.692015 26795 raft_consensus.cc:1261] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079 [term 1 FOLLOWER]: Refusing update from remote peer b821f891ad1a40d98f7ba22b338a8832: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:33:24.692014 26379 raft_consensus.cc:1261] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56 [term 1 FOLLOWER]: Refusing update from remote peer b821f891ad1a40d98f7ba22b338a8832: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:33:24.693038 26916 consensus_queue.cc:1035] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2bfc1f7bd60469d90b2db6b5a8dd079" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:33:24.693380 26911 consensus_queue.cc:1035] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e499197e9cc4c09badc59fb4882ce56" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:33:24.718798 26160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:33:24.719105 26160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:33:24.722759 26160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71add4ecf6a04b7590e7300a1a8a3387: Sending DeleteTablet for 3 replicas of tablet 7d51932cd8b741e391af0c0a2a1138a3
I20260810 23:33:24.723912 26759 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d51932cd8b741e391af0c0a2a1138a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:24 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:55964
I20260810 23:33:24.723920 26347 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d51932cd8b741e391af0c0a2a1138a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:24 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:52892
I20260810 23:33:24.724020 26544 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d51932cd8b741e391af0c0a2a1138a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:33:24 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:34892
I20260810 23:33:24.726467 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 26246
I20260810 23:33:24.736394 26934 tablet_replica.cc:330] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832: stopping tablet replica
I20260810 23:33:24.737033 26934 raft_consensus.cc:2229] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:33:24.737824 26934 raft_consensus.cc:2258] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:24.740186 26934 ts_tablet_manager.cc:1905] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:24.740851 26930 tablet_replica.cc:330] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079: stopping tablet replica
I20260810 23:33:24.741309 26930 raft_consensus.cc:2229] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:24.741886 26930 raft_consensus.cc:2258] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:24.746068 26930 ts_tablet_manager.cc:1905] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:24.747817 26931 tablet_replica.cc:330] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56: stopping tablet replica
I20260810 23:33:24.748340 26931 raft_consensus.cc:2229] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:33:24.750111 26931 raft_consensus.cc:2258] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:33:24.752099 26931 ts_tablet_manager.cc:1905] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:33:24.753080 26930 ts_tablet_manager.cc:1918] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:33:24.753274 26930 log.cc:1196] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/7d51932cd8b741e391af0c0a2a1138a3
I20260810 23:33:24.753750 26934 ts_tablet_manager.cc:1918] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:33:24.753871 26930 ts_tablet_manager.cc:1939] T 7d51932cd8b741e391af0c0a2a1138a3 P d2bfc1f7bd60469d90b2db6b5a8dd079: Deleting consensus metadata
I20260810 23:33:24.754007 26934 log.cc:1196] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/7d51932cd8b741e391af0c0a2a1138a3
I20260810 23:33:24.754612 26934 ts_tablet_manager.cc:1939] T 7d51932cd8b741e391af0c0a2a1138a3 P b821f891ad1a40d98f7ba22b338a8832: Deleting consensus metadata
I20260810 23:33:24.755967 26133 catalog_manager.cc:4841] TS d2bfc1f7bd60469d90b2db6b5a8dd079 (127.9.48.195:46493): tablet 7d51932cd8b741e391af0c0a2a1138a3 (table test-table [id=b42385a1b1814baaa1a2bcca6c99391b]) successfully deleted
I20260810 23:33:24.756469 26133 catalog_manager.cc:4841] TS b821f891ad1a40d98f7ba22b338a8832 (127.9.48.194:46173): tablet 7d51932cd8b741e391af0c0a2a1138a3 (table test-table [id=b42385a1b1814baaa1a2bcca6c99391b]) successfully deleted
Aug 10 23:33:24 dist-test-slave-04bb krb5kdc[26078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:33:24.761528 26931 ts_tablet_manager.cc:1918] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:33:24.761718 26931 log.cc:1196] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/7d51932cd8b741e391af0c0a2a1138a3
I20260810 23:33:24.762269 26931 ts_tablet_manager.cc:1939] T 7d51932cd8b741e391af0c0a2a1138a3 P 5e499197e9cc4c09badc59fb4882ce56: Deleting consensus metadata
I20260810 23:33:24.763849 26138 catalog_manager.cc:4841] TS 5e499197e9cc4c09badc59fb4882ce56 (127.9.48.193:46139): tablet 7d51932cd8b741e391af0c0a2a1138a3 (table test-table [id=b42385a1b1814baaa1a2bcca6c99391b]) successfully deleted
I20260810 23:33:24.765909 26894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.727990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.9.48.193:46139)
0810 23:33:24.728163 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:24.728169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:24.741189 (+ 13020us) server_negotiation.cc:407] Connection header received
0810 23:33:24.741528 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:24.741537 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:24.741686 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:24.741780 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:24.744118 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.745715 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.748020 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.748755 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.748878 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.758577 (+ 9699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:24.758591 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:24.758608 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:33:24.758654 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:24.762876 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:24.763754 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:24.763765 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:24.763770 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:24.763968 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:24.764457 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:24.764461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:24.764463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:24.764893 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:24.765112 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:24.765471 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 23:33:24.765627 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:33:24.766769 26440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:33:24.884649 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 26246
I20260810 23:33:24.916266 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 26452
Aug 10 23:33:24 dist-test-slave-04bb krb5kdc[26078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:33:24.933917 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:24.917136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34912 (local address 127.9.48.194:46173)
0810 23:33:24.917333 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:24.917339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:24.917363 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:33:24.917532 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:24.917538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:24.917623 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:24.917732 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:24.918928 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.920124 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.922029 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:24.922477 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:24.922578 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:24.927834 (+ 5256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:24.927850 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:24.927867 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:33:24.927901 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:24.931149 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:24.931916 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:24.931923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:24.931925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:24.932049 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:24.932453 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:24.932456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:24.932458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:24.932869 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:24.933116 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:24.933416 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 23:33:24.933593 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:33:24.934716 26641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:33:25.051136 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 26452
I20260810 23:33:25.079723 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 26662
Aug 10 23:33:25 dist-test-slave-04bb krb5kdc[26078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:33:25.095679 26942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:33:25.080489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.9.48.195:46493)
0810 23:33:25.081018 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0810 23:33:25.081023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:33:25.081040 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:33:25.081100 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:33:25.081104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:33:25.081171 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:33:25.081263 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:33:25.082582 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:25.083479 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:25.084917 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:33:25.085354 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:33:25.085482 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:33:25.089775 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:33:25.089788 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:33:25.089803 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:33:25.089835 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:33:25.093011 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:25.093696 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:25.093708 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:25.093711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:25.093891 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:33:25.094270 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:33:25.094275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:33:25.094277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:33:25.094657 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:33:25.094911 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 23:33:25.095183 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 23:33:25.095366 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":273,"threads_started":1}
I20260810 23:33:25.096333 26858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:33:25.188086 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 26662
I20260810 23:33:25.217204 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 26099
I20260810 23:33:25.218462 26199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:33:25.330060 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 26099
2026-08-10T23:33:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4989 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:33:25 dist-test-slave-04bb krb5kdc[26950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:33:25 dist-test-slave-04bb krb5kdc[26950](info): set up 2 sockets
Aug 10 23:33:25 dist-test-slave-04bb krb5kdc[26950](info): commencing operation
krb5kdc: starting...
W20260810 23:33:26.688603 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.305s user 0.004s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:33:26 dist-test-slave-04bb krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:33:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:33:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.48.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.48.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/rangeradmin_127.9.48.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/rangeradmin_127.9.48.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.48.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.48.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/rangerlookup_127.9.48.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/rangerlookup_127.9.48.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.48.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.48.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/HTTP_127.9.48.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/HTTP_127.9.48.212@KRBTEST.COM.keytab.
I20260810 23:33:27.096273 9411 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskahRJgu/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/postgres -l logfile start
2026-08-10 23:33:29.763 UTC [27005] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 23:33:29.763 UTC [27005] LOG: listening on IPv4 address "127.9.48.212", port 36947
2026-08-10 23:33:29.767 UTC [27005] LOG: listening on Unix socket "/tmp/.s.PGSQL.36947"
2026-08-10 23:33:29.790 UTC [27008] LOG: database system was shut down at 2026-08-10 23:33:28 UTC
2026-08-10 23:33:29.794 UTC [27005] LOG: database system is ready to accept connections
I20260810 23:33:30.999418 9411 mini_postgres.cc:96] Postgres bound to 36947
2026-08-10 23:33:31.027 UTC [27024] FATAL: database "slave" does not exist
127.9.48.212:36947 - accepting connections
I20260810 23:33:31.028914 9411 mini_ranger.cc:161] Starting Ranger...
I20260810 23:33:31.063555 9411 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 23:33:31.575811 9411 mini_ranger.cc:87] Created ranger Postgres database
I20260810 23:33:31.575956 9411 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-admin
2026-08-10 23:33:31,970 [I] DB FLAVOR :POSTGRES
2026-08-10 23:33:31,970 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:33:31,971 [I] Checking connection..
2026-08-10 23:33:31,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 23:33:32,241 [I] Checking connection passed.
2026-08-10 23:33:32,241 [I] --------- Verifying version history table ---------
2026-08-10 23:33:32,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:33:32,508 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:33:32,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:33:32,773 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:33:32,773 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 23:33:32,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 23:33:33.036 UTC [27132] WARNING: there is no transaction in progress
2026-08-10 23:33:33,045 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 23:33:33,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:33:33,297 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 23:33:33,298 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:33:33,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 23:33:33,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 23:33:33,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-04bb.c.gcp-upstream.internal','N') ;"
2026-08-10 23:33:34,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:33:34,348 [I] Table x_portal_user does not exist in database ranger
2026-08-10 23:33:34,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:33:34,628 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 23:33:34,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:33:34,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:33:35,139 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 23:33:35,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 23:33:36.137 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:36.153 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:36.169 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:36.184 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:36.200 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:36.218 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:36.272 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:36.279 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:36.293 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:36.302 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:36.312 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:36.326 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:36.336 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:36.343 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:36.349 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:36.356 UTC [27308] WARNING: there is no transaction in progress
2026-08-10 23:33:37,007 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 23:33:37,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 23:33:37,254 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 23:33:37,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:33:37,511 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 23:33:37,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:33:37,776 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 23:33:37,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:33:38,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:33:38,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-04bb.c.gcp-upstream.internal';"
2026-08-10 23:33:38,548 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 23:33:38,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 23:33:38,777 [I] DB_PATCHES have already been applied
I20260810 23:33:38.783705 9411 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-admin:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskahRJgu/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 23:33:38.783797 9411 mini_ranger.cc:193] Using host: 127.9.48.212
I20260810 23:33:38.806975 9411 mini_ranger.cc:227] Ranger admin URL: http://127.9.48.212:36675
Aug 10, 2026 11:33:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:33:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:33:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 11:33:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 11:33:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 11:33:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 11:33:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 23:33:40.204308 9411 mini_ranger.cc:160] Time spent starting Ranger: real 9.175s user 0.000s sys 0.068s
Aug 10, 2026 11:33:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 11:33:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 11:33:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 11:33:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 11:33:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 11:33:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 11:33:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 11:33:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 11:33:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 11:33:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 11:33:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 11:33:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 11:33:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 11:33:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 11:33:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 11:33:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 11:34:00 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 11:34:01 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 23:34:01.629346 9411 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.48.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.48.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/rangerkms_127.9.48.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/rangerkms_127.9.48.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.48.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.48.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.48.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/HTTP_127.9.48.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.48.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/HTTP_127.9.48.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 23:34:01 dist-test-slave-04bb krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404841, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 23:34:01.877619 9411 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 23:34:01.908694 9411 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 23:34:02.800850 9411 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 23:34:02.801056 9411 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-kms
I20260810 23:34:02.801199 9411 mini_ranger_kms.cc:227] Using postgres at 127.9.48.212:36947
2026-08-10 23:34:03,697 [I] DB FLAVOR :POSTGRES
2026-08-10 23:34:03,698 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:34:03,698 [I] Checking connection
2026-08-10 23:34:03,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 23:34:03,958 [I] connection success
2026-08-10 23:34:03,958 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 23:34:03,958 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 23:34:03,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 23:34:04,235 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 23:34:04,235 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:34:04,235 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 23:34:04,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:36947/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 23:34:04,503 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 23:34:04.981662 9411 mini_ranger_kms.cc:310] Created kms service
I20260810 23:34:05.164062 9411 mini_ranger_kms.cc:326] Created kudu user
I20260810 23:34:05.211477 9411 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 23:34:05 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404841, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:34:05.609580 9411 mini_ranger_kms.cc:384] Added ranger policy
I20260810 23:34:05.611263 9411 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-kms:/tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 23:34:05.611359 9411 mini_ranger_kms.cc:240] Using host: 127.9.48.212
I20260810 23:34:05.645467 9411 mini_ranger_kms.cc:276] Ranger KMS PID: 27588
I20260810 23:34:05.645666 9411 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.48.212:59413
Aug 10, 2026 11:34:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:34:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:34:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 11:34:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 11:34:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 11:34:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 23:34:07 dist-test-slave-04bb krb5kdc[26950](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786404847, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.48.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 23:34:08 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404847, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.48.212@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:34:08.596957 9411 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.719s user 0.136s sys 0.504s
I20260810 23:34:08.597342 9411 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 23:34:08.597591 9411 mini_ranger_kms.cc:401] 127.9.48.212:59413/kms/v1/keys
Aug 10 23:34:08 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404847, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.48.212@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
Aug 10 23:34:11 dist-test-slave-04bb krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:34:11.262614 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:39831
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:39831
--ranger_config_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.48.212:59413/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:34:11.618633 27678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:34:11.618963 27678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:34:11.619135 27678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:34:11.626308 27678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:34:11.626453 27678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:34:11.626559 27678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:34:11.626662 27678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:34:11.626775 27678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:34:11.636884 27678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.48.212:59413/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:39831
--ranger_config_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:39831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:34:11.638165 27678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:34:11.639480 27678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:34:11.661155 27686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:34:11.661237 27690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:34:11.661193 27694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:34:11.662062 27678 server_base.cc:1034] running on GCE node
I20260810 23:34:11.662859 27678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:34:11.664374 27678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:34:11.665606 27678 hybrid_clock.cc:648] HybridClock initialized: now 1786404851665574 us; error 72 us; skew 500 ppm
I20260810 23:34:11.665982 27678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:34:11 dist-test-slave-04bb krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404851, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:34:11.669348 27678 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:34:11.670876 27678 webserver.cc:460] Webserver started at http://127.9.48.254:45061/ using document root <none> and password file <none>
I20260810 23:34:11.671538 27678 fs_manager.cc:359] Metadata directory not provided
I20260810 23:34:11.671682 27678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:34:11.672009 27678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:34:11 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404851, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:34:11.740078 27678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "24bf30d220b84cf498d06ce03d3d3eb5"
format_stamp: "Formatted at 2026-08-10 23:34:11 on dist-test-slave-04bb"
server_key: "b8507b95e756dfdafc5a273b89912766"
server_key_iv: "3ae8e5a73d85fa1fa5859bb648ef001e"
server_key_version: "kuduclusterkey@0"
I20260810 23:34:11.741219 27678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "24bf30d220b84cf498d06ce03d3d3eb5"
format_stamp: "Formatted at 2026-08-10 23:34:11 on dist-test-slave-04bb"
server_key: "b8507b95e756dfdafc5a273b89912766"
server_key_iv: "3ae8e5a73d85fa1fa5859bb648ef001e"
server_key_version: "kuduclusterkey@0"
I20260810 23:34:11.748751 27678 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260810 23:34:11.769078 27705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:34:11.770956 27678 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260810 23:34:11.771194 27678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "24bf30d220b84cf498d06ce03d3d3eb5"
format_stamp: "Formatted at 2026-08-10 23:34:11 on dist-test-slave-04bb"
server_key: "b8507b95e756dfdafc5a273b89912766"
server_key_iv: "3ae8e5a73d85fa1fa5859bb648ef001e"
server_key_version: "kuduclusterkey@0"
I20260810 23:34:11.771471 27678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:34:11.804294 27678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:34:11.809633 27678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:34:11.809994 27678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:34:11.831094 27781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:39831 every 8 connection(s)
I20260810 23:34:11.831089 27678 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:39831
I20260810 23:34:11.833014 27678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:34:11.837090 27782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:34:11.839532 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 27678
I20260810 23:34:11.839766 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:34:11.840606 9411 external_mini_cluster.cc:1428] Setting key 927a51bfcd7cf5f0d6700d11a3bb0d4c
I20260810 23:34:11.853263 27782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5: Bootstrap starting.
I20260810 23:34:11.857867 27782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5: Neither blocks nor log segments found. Creating new log.
I20260810 23:34:11.859177 27782 log.cc:824] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5: Log is configured to *not* fsync() on all Append() calls
I20260810 23:34:11.862533 27782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5: No bootstrap required, opened a new log
Aug 10 23:34:11 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:34:11.872229 27782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bf30d220b84cf498d06ce03d3d3eb5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39831 } }
I20260810 23:34:11.872722 27782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:11.872857 27782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24bf30d220b84cf498d06ce03d3d3eb5, State: Initialized, Role: FOLLOWER
I20260810 23:34:11.873657 27782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bf30d220b84cf498d06ce03d3d3eb5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39831 } }
I20260810 23:34:11.873980 27782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:34:11.874145 27782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:34:11.874358 27782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:34:11.875200 27788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:11.842898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.9.48.254:39831)
0810 23:34:11.853166 (+ 10268us) server_negotiation.cc:206] Beginning negotiation
0810 23:34:11.853185 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:34:11.853594 (+ 409us) server_negotiation.cc:407] Connection header received
0810 23:34:11.856300 (+ 2706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:34:11.856330 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:34:11.856894 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:34:11.857332 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:34:11.858600 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:11.860449 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:11.861806 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:11.862860 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:11.863084 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:11.867382 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:34:11.867393 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:34:11.867407 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:34:11.867446 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:34:11.870658 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:11.871506 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:11.871513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:11.871519 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:11.871662 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:11.872100 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:11.872103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:11.872105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:11.872730 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:34:11.873035 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0810 23:34:11.873549 (+ 514us) server_negotiation.cc:299] Negotiation successful
0810 23:34:11.873924 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9984,"thread_start_us":9744,"threads_started":1}
I20260810 23:34:11.877408 27782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bf30d220b84cf498d06ce03d3d3eb5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39831 } }
I20260810 23:34:11.877985 27782 leader_election.cc:302] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24bf30d220b84cf498d06ce03d3d3eb5; no voters:
I20260810 23:34:11.878683 27782 leader_election.cc:288] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:34:11.878800 27790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:34:11.879535 27790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [term 1 LEADER]: Becoming Leader. State: Replica: 24bf30d220b84cf498d06ce03d3d3eb5, State: Running, Role: LEADER
I20260810 23:34:11.880173 27790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bf30d220b84cf498d06ce03d3d3eb5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39831 } }
I20260810 23:34:11.881078 27782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:34:11.883354 27794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24bf30d220b84cf498d06ce03d3d3eb5. Latest consensus state: current_term: 1 leader_uuid: "24bf30d220b84cf498d06ce03d3d3eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bf30d220b84cf498d06ce03d3d3eb5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39831 } } }
I20260810 23:34:11.883733 27794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:34:11.889158 27793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24bf30d220b84cf498d06ce03d3d3eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24bf30d220b84cf498d06ce03d3d3eb5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39831 } } }
I20260810 23:34:11.890014 27793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:34:11.891500 27803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:34:11.897475 27803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:34:11.898468 27782 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 23:34:11.918111 27803 catalog_manager.cc:1347] Generated new cluster ID: fbaa119abb024e74a002d4fd62535601
I20260810 23:34:11.918298 27803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:34:11.931026 27803 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:34:11.932479 27803 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:34:11.941713 27803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5: Generated new TSK 0
I20260810 23:34:11.942569 27803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:34:12.750047 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.48.212:59413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39831
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:34:13 dist-test-slave-04bb krb5kdc[26950](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786404853, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 23:34:13.225085 27852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:34:13.225498 27852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:34:13.225705 27852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:34:13.237670 27852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:34:13.237814 27852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:34:13.238045 27852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:34:13.253937 27852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.48.212:59413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:34:13.255858 27852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:34:13.257731 27852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:34:13.278466 27868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:34:13.278789 27874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:34:13.278877 27869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:34:13.280007 27852 server_base.cc:1034] running on GCE node
I20260810 23:34:13.280812 27852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:34:13.281932 27852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:34:13.283185 27852 hybrid_clock.cc:648] HybridClock initialized: now 1786404853283112 us; error 79 us; skew 500 ppm
I20260810 23:34:13.283550 27852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:34:13 dist-test-slave-04bb krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404853, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:34:13.292511 27852 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:34:13.299014 27852 webserver.cc:460] Webserver started at http://127.9.48.193:36711/ using document root <none> and password file <none>
I20260810 23:34:13.299845 27852 fs_manager.cc:359] Metadata directory not provided
I20260810 23:34:13.300007 27852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:34:13.300359 27852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:34:13 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404853, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:34:13.319491 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "90d198ddd38747a8a91755bb2c52aa38"
format_stamp: "Formatted at 2026-08-10 23:34:13 on dist-test-slave-04bb"
server_key: "f514fb0712c843d7b3de2a7464ad0c41"
server_key_iv: "189e090b9f4b550252fdcdb65c5f1749"
server_key_version: "kuduclusterkey@0"
I20260810 23:34:13.321514 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "90d198ddd38747a8a91755bb2c52aa38"
format_stamp: "Formatted at 2026-08-10 23:34:13 on dist-test-slave-04bb"
server_key: "f514fb0712c843d7b3de2a7464ad0c41"
server_key_iv: "189e090b9f4b550252fdcdb65c5f1749"
server_key_version: "kuduclusterkey@0"
I20260810 23:34:13.333477 27852 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.006s sys 0.005s
I20260810 23:34:13.361414 27889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:34:13.364451 27852 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.009s
I20260810 23:34:13.364765 27852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "90d198ddd38747a8a91755bb2c52aa38"
format_stamp: "Formatted at 2026-08-10 23:34:13 on dist-test-slave-04bb"
server_key: "f514fb0712c843d7b3de2a7464ad0c41"
server_key_iv: "189e090b9f4b550252fdcdb65c5f1749"
server_key_version: "kuduclusterkey@0"
I20260810 23:34:13.365476 27852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:34:13.435539 27852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:34:13.443619 27852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:34:13.444020 27852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:34:13.460981 27852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:34:13.473690 27852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:34:13.473898 27852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 23:34:13.474129 27852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:34:13.474300 27852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:34:13 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404853, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
Aug 10 23:34:13 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404853, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:34:14.079752 27852 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:35505
I20260810 23:34:14.079820 28073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:35505 every 8 connection(s)
I20260810 23:34:14.081777 27852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:34:14.091272 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 27852
I20260810 23:34:14.091487 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:34:14.092360 9411 external_mini_cluster.cc:1428] Setting key df3ed12d38e269fd99f4005e4e87266b
Aug 10 23:34:14 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404853, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
I20260810 23:34:14.154424 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:14.089754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39831 (local address 127.9.48.193:49815)
0810 23:34:14.102959 (+ 13205us) negotiation.cc:107] Waiting for socket to connect
0810 23:34:14.103057 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0810 23:34:14.106130 (+ 3073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:34:14.122806 (+ 16676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:34:14.122827 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:34:14.123600 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:34:14.124294 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:14.124353 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:14.125976 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:14.125982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:14.127960 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:14.127977 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:14.129111 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:14.129118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:14.129414 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:14.140988 (+ 11574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:34:14.141028 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:34:14.144133 (+ 3105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:34:14.148726 (+ 4593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:34:14.148744 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:34:14.148770 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:34:14.149586 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:34:14.150161 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:34:14.150166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:34:14.150169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:34:14.150464 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:34:14.151195 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:34:14.151204 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:34:14.151745 (+ 541us) client_negotiation.cc:769] Sending connection context
0810 23:34:14.152126 (+ 381us) client_negotiation.cc:240] Negotiation successful
0810 23:34:14.152827 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12726,"thread_start_us":12420,"threads_started":1}
I20260810 23:34:14.156059 28078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:14.089100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:49815 (local address 127.9.48.254:39831)
0810 23:34:14.121967 (+ 32867us) server_negotiation.cc:206] Beginning negotiation
0810 23:34:14.121975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:34:14.122000 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:34:14.122078 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:34:14.122084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:34:14.122176 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:34:14.122343 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:34:14.124560 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:14.125731 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:14.128209 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:14.128848 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:14.129849 (+ 1001us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:14.144501 (+ 14652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:34:14.144510 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:34:14.144515 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:34:14.144588 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:34:14.148440 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:14.149784 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:14.149801 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:14.149805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:14.149974 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:14.150634 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:14.150638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:14.150641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:14.151022 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:34:14.155268 (+ 4246us) server_negotiation.cc:1035] Waiting for connection context
0810 23:34:14.155411 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 23:34:14.155662 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32700,"thread_start_us":33386,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
I20260810 23:34:14.167160 28074 heartbeater.cc:347] Connected to a master server at 127.9.48.254:39831
I20260810 23:34:14.167599 28074 heartbeater.cc:464] Registering TS with master...
I20260810 23:34:14.168673 28074 heartbeater.cc:511] Master 127.9.48.254:39831 requested a full tablet report, sending...
I20260810 23:34:14.171561 27729 ts_manager.cc:194] Registered new tserver with Master: 90d198ddd38747a8a91755bb2c52aa38 (127.9.48.193:35505)
I20260810 23:34:14.173633 27729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:49815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
W20260810 23:34:14.206971 9411 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.9.48.194: real 0.111s user 0.000s sys 0.027s
I20260810 23:34:14.207897 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.48.212:59413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39831
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:34:14.665024 28086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:34:14.665408 28086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:34:14.665624 28086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:34:14.674551 28086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:34:14.674726 28086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:34:14.674911 28086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:34:14.686096 28086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.48.212:59413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:34:14.687508 28086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:34:14.689162 28086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:34:14.712775 28108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:34:14.713475 28110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:34:14.713585 28112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:34:14.715304 28086 server_base.cc:1034] running on GCE node
I20260810 23:34:14.716058 28086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:34:14.717229 28086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:34:14.718524 28086 hybrid_clock.cc:648] HybridClock initialized: now 1786404854718450 us; error 38 us; skew 500 ppm
I20260810 23:34:14.718924 28086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:34:14 dist-test-slave-04bb krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404854, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:34:14.722760 28086 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:34:14.724459 28086 webserver.cc:460] Webserver started at http://127.9.48.194:44859/ using document root <none> and password file <none>
I20260810 23:34:14.725369 28086 fs_manager.cc:359] Metadata directory not provided
I20260810 23:34:14.725525 28086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:34:14.725893 28086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:34:14 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404854, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:34:14.743937 28086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "7e79f4066a0a48f09485983c78d843cb"
format_stamp: "Formatted at 2026-08-10 23:34:14 on dist-test-slave-04bb"
server_key: "1405e5440444b25a2cf665e138e71d0c"
server_key_iv: "1d97840ad4431d236c2386cd8dc30cab"
server_key_version: "kuduclusterkey@0"
I20260810 23:34:14.745491 28086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "7e79f4066a0a48f09485983c78d843cb"
format_stamp: "Formatted at 2026-08-10 23:34:14 on dist-test-slave-04bb"
server_key: "1405e5440444b25a2cf665e138e71d0c"
server_key_iv: "1d97840ad4431d236c2386cd8dc30cab"
server_key_version: "kuduclusterkey@0"
I20260810 23:34:14.753228 28086 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260810 23:34:14.773604 28124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:34:14.775650 28086 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260810 23:34:14.775882 28086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "7e79f4066a0a48f09485983c78d843cb"
format_stamp: "Formatted at 2026-08-10 23:34:14 on dist-test-slave-04bb"
server_key: "1405e5440444b25a2cf665e138e71d0c"
server_key_iv: "1d97840ad4431d236c2386cd8dc30cab"
server_key_version: "kuduclusterkey@0"
I20260810 23:34:14.776185 28086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:34:14.795480 28086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:34:14.801172 28086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:34:14.801645 28086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:34:14.802984 28086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:34:14.805116 28086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:34:14.805223 28086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:34:14.805373 28086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:34:14.805491 28086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:34:14.871811 28086 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:37029
I20260810 23:34:14.871871 28304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:37029 every 8 connection(s)
I20260810 23:34:14.873956 28086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:34:14.877235 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 28086
I20260810 23:34:14.877422 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:34:14.878096 9411 external_mini_cluster.cc:1428] Setting key 3e2fcf6e2e6e987006dc4fcb12cd3726
Aug 10 23:34:14 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404854, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
I20260810 23:34:14.914731 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:14.877514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:33015 (local address 127.9.48.254:39831)
0810 23:34:14.889027 (+ 11513us) server_negotiation.cc:206] Beginning negotiation
0810 23:34:14.889037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:34:14.892431 (+ 3394us) server_negotiation.cc:407] Connection header received
0810 23:34:14.892493 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:34:14.892497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:34:14.892619 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:34:14.892737 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:34:14.894388 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:14.895740 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:14.899053 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:14.899866 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:14.900027 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:14.905654 (+ 5627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:34:14.905663 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:34:14.905671 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:34:14.905719 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:34:14.910341 (+ 4622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:14.911586 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:14.911595 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:14.911600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:14.911777 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:14.912383 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:14.912389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:14.912394 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:14.912894 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:34:14.913122 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 23:34:14.914109 (+ 987us) server_negotiation.cc:299] Negotiation successful
0810 23:34:14.914307 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11307,"thread_start_us":11157,"threads_started":1}
I20260810 23:34:14.916478 28311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:14.878274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39831 (local address 127.9.48.194:33015)
0810 23:34:14.889748 (+ 11474us) negotiation.cc:107] Waiting for socket to connect
0810 23:34:14.889816 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 23:34:14.892067 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:34:14.893045 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:34:14.893053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:34:14.893522 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:34:14.894204 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:14.894224 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:14.896011 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:14.896017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:14.898632 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:14.898661 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:14.900199 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:14.900210 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:14.900575 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:14.901546 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:34:14.901581 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:34:14.905327 (+ 3746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:34:14.910526 (+ 5199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:34:14.910537 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:34:14.910560 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:34:14.911414 (+ 854us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:34:14.911929 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:34:14.911935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:34:14.911939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:34:14.912232 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:34:14.913075 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:34:14.913085 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:34:14.913552 (+ 467us) client_negotiation.cc:769] Sending connection context
0810 23:34:14.913914 (+ 362us) client_negotiation.cc:240] Negotiation successful
0810 23:34:14.914772 (+ 858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11086,"thread_start_us":10802,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
I20260810 23:34:14.918486 28305 heartbeater.cc:347] Connected to a master server at 127.9.48.254:39831
I20260810 23:34:14.918884 28305 heartbeater.cc:464] Registering TS with master...
I20260810 23:34:14.919881 28305 heartbeater.cc:511] Master 127.9.48.254:39831 requested a full tablet report, sending...
I20260810 23:34:14.921886 27729 ts_manager.cc:194] Registered new tserver with Master: 7e79f4066a0a48f09485983c78d843cb (127.9.48.194:37029)
I20260810 23:34:14.923193 27729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:33015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:34:14.952036 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.48.212:59413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39831
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:34:15.178092 28074 heartbeater.cc:503] Master 127.9.48.254:39831 was elected leader, sending a full tablet report...
W20260810 23:34:15.384097 28315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:34:15.384404 28315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:34:15.384510 28315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:34:15.392967 28315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:34:15.393112 28315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:34:15.393314 28315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:34:15.404428 28315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.48.212:59413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:34:15.405845 28315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:34:15.407315 28315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:34:15.422704 28326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:34:15.423012 28323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:34:15.423074 28329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:34:15.423951 28315 server_base.cc:1034] running on GCE node
I20260810 23:34:15.424685 28315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:34:15.425803 28315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:34:15.427066 28315 hybrid_clock.cc:648] HybridClock initialized: now 1786404855426987 us; error 51 us; skew 500 ppm
I20260810 23:34:15.427451 28315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:34:15 dist-test-slave-04bb krb5kdc[26950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404855, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:34:15.431211 28315 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:34:15.432919 28315 webserver.cc:460] Webserver started at http://127.9.48.195:37819/ using document root <none> and password file <none>
I20260810 23:34:15.433604 28315 fs_manager.cc:359] Metadata directory not provided
I20260810 23:34:15.433738 28315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:34:15.434057 28315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:34:15 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404855, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:34:15.451318 28315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "cb4ee26da43642799f96cc8c59f82e27"
format_stamp: "Formatted at 2026-08-10 23:34:15 on dist-test-slave-04bb"
server_key: "f6bf64ba8cd853e01e32320a25cd3021"
server_key_iv: "3cc08f2b3e4936d199f595d376bba3c1"
server_key_version: "kuduclusterkey@0"
I20260810 23:34:15.452291 28315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "cb4ee26da43642799f96cc8c59f82e27"
format_stamp: "Formatted at 2026-08-10 23:34:15 on dist-test-slave-04bb"
server_key: "f6bf64ba8cd853e01e32320a25cd3021"
server_key_iv: "3cc08f2b3e4936d199f595d376bba3c1"
server_key_version: "kuduclusterkey@0"
I20260810 23:34:15.458793 28315 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260810 23:34:15.475071 28344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:34:15.476835 28315 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 23:34:15.477053 28315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "cb4ee26da43642799f96cc8c59f82e27"
format_stamp: "Formatted at 2026-08-10 23:34:15 on dist-test-slave-04bb"
server_key: "f6bf64ba8cd853e01e32320a25cd3021"
server_key_iv: "3cc08f2b3e4936d199f595d376bba3c1"
server_key_version: "kuduclusterkey@0"
I20260810 23:34:15.477362 28315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:34:15.498955 28315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:34:15.503404 28315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:34:15.503782 28315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:34:15.504997 28315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:34:15.506721 28315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:34:15.506841 28315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:34:15.507035 28315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:34:15.507131 28315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:34:15.555310 28315 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:35031
I20260810 23:34:15.555354 28545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:35031 every 8 connection(s)
I20260810 23:34:15.556895 28315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:34:15.564189 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 28315
I20260810 23:34:15.564410 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:34:15.564934 9411 external_mini_cluster.cc:1428] Setting key dc954e90a6f279ca341818200fe71a0b
Aug 10 23:34:15 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404855, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:34:15.579604 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:15.559854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:59551 (local address 127.9.48.254:39831)
0810 23:34:15.560433 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0810 23:34:15.560438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:34:15.561435 (+ 997us) server_negotiation.cc:407] Connection header received
0810 23:34:15.563443 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:34:15.563448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:34:15.563576 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:34:15.563698 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:34:15.565486 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.566468 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.568421 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.569064 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.569212 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:15.573087 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:34:15.573093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:34:15.573096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:34:15.573135 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:34:15.576071 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:15.577018 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:15.577022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:15.577024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:15.577187 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:15.577711 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:15.577721 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:15.577723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:15.578025 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:34:15.578211 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 23:34:15.578918 (+ 707us) server_negotiation.cc:299] Negotiation successful
0810 23:34:15.579171 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":337,"threads_started":1}
I20260810 23:34:15.580787 28552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:15.560402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39831 (local address 127.9.48.195:59551)
0810 23:34:15.561252 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0810 23:34:15.561301 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:34:15.563145 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:34:15.563993 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:34:15.564002 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:34:15.564612 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:34:15.565300 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:15.565325 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.566621 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.566625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:15.568251 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:15.568263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.569371 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.569382 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:15.569642 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:15.570385 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:34:15.570408 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:34:15.572822 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:34:15.576220 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:34:15.576229 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:34:15.576245 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:34:15.576862 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:34:15.577305 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:34:15.577311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:34:15.577316 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:34:15.577569 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:34:15.578163 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:34:15.578171 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:34:15.578534 (+ 363us) client_negotiation.cc:769] Sending connection context
0810 23:34:15.578799 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 23:34:15.579367 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":369,"threads_started":1}
I20260810 23:34:15.582563 28547 heartbeater.cc:347] Connected to a master server at 127.9.48.254:39831
I20260810 23:34:15.582947 28547 heartbeater.cc:464] Registering TS with master...
I20260810 23:34:15.583850 28547 heartbeater.cc:511] Master 127.9.48.254:39831 requested a full tablet report, sending...
I20260810 23:34:15.585489 27729 ts_manager.cc:194] Registered new tserver with Master: cb4ee26da43642799f96cc8c59f82e27 (127.9.48.195:35031)
I20260810 23:34:15.586447 27729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:59551
I20260810 23:34:15.593590 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:34:15.611732 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:15.599049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.9.48.254:39831)
0810 23:34:15.599221 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 23:34:15.599225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:34:15.599783 (+ 558us) server_negotiation.cc:407] Connection header received
0810 23:34:15.599944 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:34:15.599949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:34:15.600025 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:34:15.600112 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:34:15.601722 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.602490 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.603736 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.604174 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.604282 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:15.606731 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:34:15.606736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:34:15.606740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:34:15.606772 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:34:15.609453 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:15.610073 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:15.610077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:15.610080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:15.610206 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:15.610558 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:15.610561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:15.610564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:15.610836 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:34:15.611025 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 23:34:15.611309 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 23:34:15.611479 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:34:15.615868 27729 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:34:15.624079 27729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:34:15.690910 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:15.642831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54986 (local address 127.9.48.193:35505)
0810 23:34:15.676881 (+ 34050us) server_negotiation.cc:206] Beginning negotiation
0810 23:34:15.676897 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:34:15.681582 (+ 4685us) server_negotiation.cc:407] Connection header received
0810 23:34:15.681928 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:34:15.681936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:34:15.682213 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:34:15.682440 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:34:15.684093 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.685534 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.688440 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.689572 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.689733 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:15.690001 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 23:34:15.690283 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 23:34:15.690469 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33763,"thread_start_us":34073,"threads_started":1}
I20260810 23:34:15.691162 28571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:15.641971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:35505 (local address 127.0.0.1:54986)
0810 23:34:15.681449 (+ 39478us) negotiation.cc:107] Waiting for socket to connect
0810 23:34:15.681469 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:34:15.681700 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:34:15.683051 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:34:15.683056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:34:15.683115 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:34:15.683824 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:15.683838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.685686 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.685692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:15.688218 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:15.688234 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.689815 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.689822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:15.690037 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:15.690095 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 23:34:15.690238 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 23:34:15.690437 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39313,"spinlock_wait_cycles":1280,"thread_start_us":35300,"threads_started":1}
I20260810 23:34:15.692566 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:15.638738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.9.48.194:37029)
0810 23:34:15.678950 (+ 40212us) server_negotiation.cc:206] Beginning negotiation
0810 23:34:15.678964 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:34:15.680155 (+ 1191us) server_negotiation.cc:407] Connection header received
0810 23:34:15.680589 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:34:15.680601 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:34:15.680973 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:34:15.681218 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:34:15.682335 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.684638 (+ 2303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.688344 (+ 3706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.690260 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.690430 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:15.690643 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 23:34:15.691969 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0810 23:34:15.692148 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39769,"thread_start_us":40216,"threads_started":1}
I20260810 23:34:15.693275 28578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:15.638509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.9.48.195:35031)
0810 23:34:15.681734 (+ 43225us) server_negotiation.cc:206] Beginning negotiation
0810 23:34:15.681744 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:34:15.681837 (+ 93us) server_negotiation.cc:407] Connection header received
0810 23:34:15.681938 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:34:15.681943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:34:15.682371 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:34:15.682602 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:34:15.683938 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.685763 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.688652 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.690418 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.690540 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:15.692468 (+ 1928us) server_negotiation.cc:1035] Waiting for connection context
0810 23:34:15.692618 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 23:34:15.692812 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43020,"thread_start_us":42244,"threads_started":1}
I20260810 23:34:15.693936 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:15.638204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:35031 (local address 127.0.0.1:36394)
0810 23:34:15.675482 (+ 37278us) negotiation.cc:107] Waiting for socket to connect
0810 23:34:15.675559 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0810 23:34:15.675884 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:34:15.682840 (+ 6956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:34:15.682848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:34:15.682877 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:34:15.683622 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:15.683637 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.685931 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.685942 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:15.688477 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:15.688490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.690925 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.690931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:15.691111 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:15.691152 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 23:34:15.691249 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:34:15.691398 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36799,"mutex_wait_us":53,"thread_start_us":42919,"threads_started":1}
I20260810 23:34:15.693951 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:15.638506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:37029 (local address 127.0.0.1:54504)
0810 23:34:15.679986 (+ 41480us) negotiation.cc:107] Waiting for socket to connect
0810 23:34:15.680015 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 23:34:15.680312 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:34:15.681242 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:34:15.681252 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:34:15.681325 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:34:15.681916 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:15.681933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.684811 (+ 2878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.684817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:15.688166 (+ 3349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:15.688178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.690868 (+ 2690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.690883 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:15.691773 (+ 890us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:15.691807 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:34:15.691911 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 23:34:15.692068 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41287,"mutex_wait_us":4,"thread_start_us":41086,"threads_started":1}
I20260810 23:34:15.696272 27967 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba160a3874742e483a29c05815fd7b6 (DEFAULT_TABLE table=test-table [id=4dd19b56e15142f98d4df23f455e0a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:34:15.698259 27967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba160a3874742e483a29c05815fd7b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:34:15.698132 28199 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba160a3874742e483a29c05815fd7b6 (DEFAULT_TABLE table=test-table [id=4dd19b56e15142f98d4df23f455e0a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:34:15.699829 28199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba160a3874742e483a29c05815fd7b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:34:15.703861 28435 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba160a3874742e483a29c05815fd7b6 (DEFAULT_TABLE table=test-table [id=4dd19b56e15142f98d4df23f455e0a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:34:15.705379 28435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba160a3874742e483a29c05815fd7b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:34:15.716827 28584 tablet_bootstrap.cc:492] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27: Bootstrap starting.
I20260810 23:34:15.720511 28584 tablet_bootstrap.cc:654] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27: Neither blocks nor log segments found. Creating new log.
I20260810 23:34:15.721887 28586 tablet_bootstrap.cc:492] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38: Bootstrap starting.
I20260810 23:34:15.723224 28584 log.cc:824] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27: Log is configured to *not* fsync() on all Append() calls
I20260810 23:34:15.725173 28591 tablet_bootstrap.cc:492] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb: Bootstrap starting.
I20260810 23:34:15.726584 28586 tablet_bootstrap.cc:654] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38: Neither blocks nor log segments found. Creating new log.
I20260810 23:34:15.727154 28584 tablet_bootstrap.cc:492] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27: No bootstrap required, opened a new log
I20260810 23:34:15.727484 28584 ts_tablet_manager.cc:1397] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260810 23:34:15.728236 28586 log.cc:824] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38: Log is configured to *not* fsync() on all Append() calls
I20260810 23:34:15.729559 28591 tablet_bootstrap.cc:654] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb: Neither blocks nor log segments found. Creating new log.
I20260810 23:34:15.731052 28591 log.cc:824] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb: Log is configured to *not* fsync() on all Append() calls
I20260810 23:34:15.731995 28586 tablet_bootstrap.cc:492] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38: No bootstrap required, opened a new log
I20260810 23:34:15.732326 28586 ts_tablet_manager.cc:1397] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260810 23:34:15.735707 28591 tablet_bootstrap.cc:492] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb: No bootstrap required, opened a new log
I20260810 23:34:15.736011 28591 ts_tablet_manager.cc:1397] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260810 23:34:15.738106 28584 raft_consensus.cc:348] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4ee26da43642799f96cc8c59f82e27" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35031 } } peers { permanent_uuid: "7e79f4066a0a48f09485983c78d843cb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37029 } } peers { permanent_uuid: "90d198ddd38747a8a91755bb2c52aa38" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 35505 } }
I20260810 23:34:15.738566 28584 raft_consensus.cc:374] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:15.738722 28584 raft_consensus.cc:729] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb4ee26da43642799f96cc8c59f82e27, State: Initialized, Role: FOLLOWER
I20260810 23:34:15.739470 28584 consensus_queue.cc:260] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4ee26da43642799f96cc8c59f82e27" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35031 } } peers { permanent_uuid: "7e79f4066a0a48f09485983c78d843cb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37029 } } peers { permanent_uuid: "90d198ddd38747a8a91755bb2c52aa38" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 35505 } }
I20260810 23:34:15.740890 28547 heartbeater.cc:503] Master 127.9.48.254:39831 was elected leader, sending a full tablet report...
I20260810 23:34:15.741679 28584 ts_tablet_manager.cc:1428] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260810 23:34:15.744095 28586 raft_consensus.cc:348] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4ee26da43642799f96cc8c59f82e27" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35031 } } peers { permanent_uuid: "7e79f4066a0a48f09485983c78d843cb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37029 } } peers { permanent_uuid: "90d198ddd38747a8a91755bb2c52aa38" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 35505 } }
I20260810 23:34:15.744668 28586 raft_consensus.cc:374] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:15.744872 28586 raft_consensus.cc:729] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d198ddd38747a8a91755bb2c52aa38, State: Initialized, Role: FOLLOWER
I20260810 23:34:15.745705 28586 consensus_queue.cc:260] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4ee26da43642799f96cc8c59f82e27" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35031 } } peers { permanent_uuid: "7e79f4066a0a48f09485983c78d843cb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37029 } } peers { permanent_uuid: "90d198ddd38747a8a91755bb2c52aa38" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 35505 } }
I20260810 23:34:15.747661 28586 ts_tablet_manager.cc:1428] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260810 23:34:15.747524 28591 raft_consensus.cc:348] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4ee26da43642799f96cc8c59f82e27" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35031 } } peers { permanent_uuid: "7e79f4066a0a48f09485983c78d843cb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37029 } } peers { permanent_uuid: "90d198ddd38747a8a91755bb2c52aa38" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 35505 } }
I20260810 23:34:15.747997 28591 raft_consensus.cc:374] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:34:15.748107 28591 raft_consensus.cc:729] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e79f4066a0a48f09485983c78d843cb, State: Initialized, Role: FOLLOWER
I20260810 23:34:15.748769 28591 consensus_queue.cc:260] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4ee26da43642799f96cc8c59f82e27" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35031 } } peers { permanent_uuid: "7e79f4066a0a48f09485983c78d843cb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37029 } } peers { permanent_uuid: "90d198ddd38747a8a91755bb2c52aa38" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 35505 } }
I20260810 23:34:15.749886 28305 heartbeater.cc:503] Master 127.9.48.254:39831 was elected leader, sending a full tablet report...
I20260810 23:34:15.750372 28591 ts_tablet_manager.cc:1428] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 23:34:15.786273 28603 raft_consensus.cc:482] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:34:15.786564 28603 raft_consensus.cc:504] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4ee26da43642799f96cc8c59f82e27" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35031 } } peers { permanent_uuid: "7e79f4066a0a48f09485983c78d843cb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37029 } } peers { permanent_uuid: "90d198ddd38747a8a91755bb2c52aa38" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 35505 } }
I20260810 23:34:15.788331 28603 leader_election.cc:288] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb4ee26da43642799f96cc8c59f82e27 (127.9.48.195:35031), 90d198ddd38747a8a91755bb2c52aa38 (127.9.48.193:35505)
I20260810 23:34:15.797700 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:15.789613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:48681 (local address 127.9.48.193:35505)
0810 23:34:15.789854 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:34:15.789860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:34:15.789876 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:34:15.789943 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:34:15.789947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:34:15.790015 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:34:15.790107 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:34:15.791028 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.792244 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.795244 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.796563 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.796721 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:15.796790 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:34:15.797314 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 23:34:15.797444 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:34:15.797698 28578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:15.788889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:39449 (local address 127.9.48.195:35031)
0810 23:34:15.789338 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0810 23:34:15.789348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:34:15.789541 (+ 193us) server_negotiation.cc:407] Connection header received
0810 23:34:15.789900 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:34:15.789907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:34:15.789991 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:34:15.790097 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:34:15.790960 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.792176 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.795251 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.796553 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.796716 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:15.796790 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:34:15.797321 (+ 531us) server_negotiation.cc:299] Negotiation successful
0810 23:34:15.797444 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
I20260810 23:34:15.797930 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:15.788576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:35031 (local address 127.9.48.194:39449)
0810 23:34:15.789405 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0810 23:34:15.789433 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:34:15.789683 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:34:15.790146 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:34:15.790151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:34:15.790181 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:34:15.790780 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:15.790800 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.792337 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.792344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:15.795043 (+ 2699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:15.795059 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.796845 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.796851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:15.797094 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:15.797172 (+ 78us) client_negotiation.cc:769] Sending connection context
0810 23:34:15.797270 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 23:34:15.797425 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":269,"thread_start_us":405,"threads_started":1}
I20260810 23:34:15.797962 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:15.788634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:35505 (local address 127.9.48.194:48681)
0810 23:34:15.789702 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0810 23:34:15.789718 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:34:15.789824 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:34:15.790173 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:34:15.790177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:34:15.790194 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:34:15.790801 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:15.790810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.792370 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.792377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:15.795043 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:34:15.795059 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.796830 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.796837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:34:15.797092 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:15.797166 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 23:34:15.797270 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 23:34:15.797425 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":223,"thread_start_us":700,"threads_started":1}
I20260810 23:34:15.798940 28473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba160a3874742e483a29c05815fd7b6" candidate_uuid: "7e79f4066a0a48f09485983c78d843cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb4ee26da43642799f96cc8c59f82e27" is_pre_election: true
I20260810 23:34:15.798998 27996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba160a3874742e483a29c05815fd7b6" candidate_uuid: "7e79f4066a0a48f09485983c78d843cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d198ddd38747a8a91755bb2c52aa38" is_pre_election: true
I20260810 23:34:15.799467 28473 raft_consensus.cc:2454] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e79f4066a0a48f09485983c78d843cb in term 0.
I20260810 23:34:15.799546 27996 raft_consensus.cc:2454] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e79f4066a0a48f09485983c78d843cb in term 0.
I20260810 23:34:15.800166 28126 leader_election.cc:302] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e79f4066a0a48f09485983c78d843cb, cb4ee26da43642799f96cc8c59f82e27; no voters:
I20260810 23:34:15.800594 28603 raft_consensus.cc:2781] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:34:15.800755 28603 raft_consensus.cc:482] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:34:15.800871 28603 raft_consensus.cc:3037] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:34:15.803339 28603 raft_consensus.cc:504] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4ee26da43642799f96cc8c59f82e27" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35031 } } peers { permanent_uuid: "7e79f4066a0a48f09485983c78d843cb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37029 } } peers { permanent_uuid: "90d198ddd38747a8a91755bb2c52aa38" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 35505 } }
I20260810 23:34:15.804008 28603 leader_election.cc:288] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [CANDIDATE]: Term 1 election: Requested vote from peers cb4ee26da43642799f96cc8c59f82e27 (127.9.48.195:35031), 90d198ddd38747a8a91755bb2c52aa38 (127.9.48.193:35505)
I20260810 23:34:15.804445 28473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba160a3874742e483a29c05815fd7b6" candidate_uuid: "7e79f4066a0a48f09485983c78d843cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb4ee26da43642799f96cc8c59f82e27"
I20260810 23:34:15.804697 28473 raft_consensus.cc:3037] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:34:15.804684 27996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba160a3874742e483a29c05815fd7b6" candidate_uuid: "7e79f4066a0a48f09485983c78d843cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d198ddd38747a8a91755bb2c52aa38"
I20260810 23:34:15.804908 27996 raft_consensus.cc:3037] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:34:15.807189 28473 raft_consensus.cc:2454] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e79f4066a0a48f09485983c78d843cb in term 1.
I20260810 23:34:15.807264 27996 raft_consensus.cc:2454] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e79f4066a0a48f09485983c78d843cb in term 1.
I20260810 23:34:15.807637 28126 leader_election.cc:302] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e79f4066a0a48f09485983c78d843cb, cb4ee26da43642799f96cc8c59f82e27; no voters:
I20260810 23:34:15.807986 28603 raft_consensus.cc:2781] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:34:15.808410 28603 raft_consensus.cc:686] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [term 1 LEADER]: Becoming Leader. State: Replica: 7e79f4066a0a48f09485983c78d843cb, State: Running, Role: LEADER
W20260810 23:34:15.808982 28548 tablet.cc:2369] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:34:15.808920 28603 consensus_queue.cc:237] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4ee26da43642799f96cc8c59f82e27" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35031 } } peers { permanent_uuid: "7e79f4066a0a48f09485983c78d843cb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37029 } } peers { permanent_uuid: "90d198ddd38747a8a91755bb2c52aa38" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 35505 } }
I20260810 23:34:15.813993 27727 catalog_manager.cc:5495] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e79f4066a0a48f09485983c78d843cb (127.9.48.194). New cstate: current_term: 1 leader_uuid: "7e79f4066a0a48f09485983c78d843cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb4ee26da43642799f96cc8c59f82e27" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e79f4066a0a48f09485983c78d843cb" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d198ddd38747a8a91755bb2c52aa38" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 35505 } health_report { overall_health: UNKNOWN } } }
I20260810 23:34:15.849879 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:15.843754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.9.48.194:37029)
0810 23:34:15.843988 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 23:34:15.843995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:34:15.844020 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:34:15.844139 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:34:15.844145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:34:15.844264 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:34:15.844425 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:34:15.844818 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.845940 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.847386 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.848027 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.848152 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:15.848674 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:34:15.849347 (+ 673us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:34:15.849465 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:34:15.849572 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 23:34:15.849661 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:34:15.858641 28473 raft_consensus.cc:1261] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27 [term 1 FOLLOWER]: Refusing update from remote peer 7e79f4066a0a48f09485983c78d843cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:34:15.858630 27996 raft_consensus.cc:1261] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38 [term 1 FOLLOWER]: Refusing update from remote peer 7e79f4066a0a48f09485983c78d843cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:34:15.859803 28608 consensus_queue.cc:1035] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb4ee26da43642799f96cc8c59f82e27" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:34:15.860160 28603 consensus_queue.cc:1035] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d198ddd38747a8a91755bb2c52aa38" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 35505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:34:15.873857 28613 mvcc.cc:205] Tried to move back new op lower bound from 7317114289585672192 to 7317114289397833728. Current Snapshot: MvccSnapshot[applied={T|T < 7317114289585672192}]
W20260810 23:34:15.874889 28075 tablet.cc:2369] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:34:15.875309 28615 mvcc.cc:205] Tried to move back new op lower bound from 7317114289585672192 to 7317114289397833728. Current Snapshot: MvccSnapshot[applied={T|T < 7317114289585672192}]
I20260810 23:34:15.904762 27727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:34:15.905072 27727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:34:15.908710 27727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24bf30d220b84cf498d06ce03d3d3eb5: Sending DeleteTablet for 3 replicas of tablet 9ba160a3874742e483a29c05815fd7b6
I20260810 23:34:15.910094 28199 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba160a3874742e483a29c05815fd7b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:34:15 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:54504
I20260810 23:34:15.910233 27967 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba160a3874742e483a29c05815fd7b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:34:15 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:54986
I20260810 23:34:15.910207 28435 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba160a3874742e483a29c05815fd7b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:34:15 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:36394
I20260810 23:34:15.914541 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 27852
I20260810 23:34:15.922420 28629 tablet_replica.cc:330] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb: stopping tablet replica
I20260810 23:34:15.922896 28629 raft_consensus.cc:2229] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:34:15.923730 28629 raft_consensus.cc:2258] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:34:15.926159 28629 ts_tablet_manager.cc:1905] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:34:15.933885 28629 ts_tablet_manager.cc:1918] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:34:15.934110 28629 log.cc:1196] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/9ba160a3874742e483a29c05815fd7b6
I20260810 23:34:15.934755 28629 ts_tablet_manager.cc:1939] T 9ba160a3874742e483a29c05815fd7b6 P 7e79f4066a0a48f09485983c78d843cb: Deleting consensus metadata
I20260810 23:34:15.936991 27710 catalog_manager.cc:4841] TS 7e79f4066a0a48f09485983c78d843cb (127.9.48.194:37029): tablet 9ba160a3874742e483a29c05815fd7b6 (table test-table [id=4dd19b56e15142f98d4df23f455e0a94]) successfully deleted
I20260810 23:34:15.941104 28631 tablet_replica.cc:330] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38: stopping tablet replica
I20260810 23:34:15.941757 28631 raft_consensus.cc:2229] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:34:15.942407 28631 raft_consensus.cc:2258] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:34:15.943326 28633 tablet_replica.cc:330] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27: stopping tablet replica
I20260810 23:34:15.943770 28633 raft_consensus.cc:2229] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:34:15.944278 28633 raft_consensus.cc:2258] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:34:15.945652 28631 ts_tablet_manager.cc:1905] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:34:15.946223 28633 ts_tablet_manager.cc:1905] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 23:34:15 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:34:15.953001 28631 ts_tablet_manager.cc:1918] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:34:15.953259 28633 ts_tablet_manager.cc:1918] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:34:15.953296 28631 log.cc:1196] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/9ba160a3874742e483a29c05815fd7b6
I20260810 23:34:15.953478 28633 log.cc:1196] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/9ba160a3874742e483a29c05815fd7b6
I20260810 23:34:15.953966 28631 ts_tablet_manager.cc:1939] T 9ba160a3874742e483a29c05815fd7b6 P 90d198ddd38747a8a91755bb2c52aa38: Deleting consensus metadata
I20260810 23:34:15.954007 28633 ts_tablet_manager.cc:1939] T 9ba160a3874742e483a29c05815fd7b6 P cb4ee26da43642799f96cc8c59f82e27: Deleting consensus metadata
I20260810 23:34:15.955921 27706 catalog_manager.cc:4841] TS cb4ee26da43642799f96cc8c59f82e27 (127.9.48.195:35031): tablet 9ba160a3874742e483a29c05815fd7b6 (table test-table [id=4dd19b56e15142f98d4df23f455e0a94]) successfully deleted
I20260810 23:34:15.956413 27709 catalog_manager.cc:4841] TS 90d198ddd38747a8a91755bb2c52aa38 (127.9.48.193:35505): tablet 9ba160a3874742e483a29c05815fd7b6 (table test-table [id=4dd19b56e15142f98d4df23f455e0a94]) successfully deleted
I20260810 23:34:15.960467 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:15.916381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54998 (local address 127.9.48.193:35505)
0810 23:34:15.916631 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 23:34:15.916640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:34:15.939554 (+ 22914us) server_negotiation.cc:407] Connection header received
0810 23:34:15.939868 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:34:15.939879 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:34:15.940017 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:34:15.940115 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:34:15.942661 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.943688 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.945700 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:15.946390 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:15.946491 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:15.952351 (+ 5860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:34:15.952377 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:34:15.952395 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:34:15.952447 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:34:15.956928 (+ 4481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:15.957726 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:15.957737 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:15.957741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:15.957947 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:15.958505 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:15.958513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:15.958516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:15.959135 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:34:15.959501 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0810 23:34:15.959852 (+ 351us) server_negotiation.cc:299] Negotiation successful
0810 23:34:15.960086 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:34:15.961257 28064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:34:16.055136 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 27852
I20260810 23:34:16.085602 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 28086
Aug 10 23:34:16 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:34:16.105894 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:16.086440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.9.48.194:37029)
0810 23:34:16.086639 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:34:16.086644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:34:16.086711 (+ 67us) server_negotiation.cc:407] Connection header received
0810 23:34:16.086966 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:34:16.086971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:34:16.087044 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:34:16.087188 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:34:16.088942 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:16.089804 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:16.091876 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:16.092675 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:16.092833 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:16.099469 (+ 6636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:34:16.099483 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:34:16.099500 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:34:16.099537 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:34:16.102947 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:16.103750 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:16.103757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:16.103760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:16.103974 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:16.104375 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:16.104378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:16.104380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:16.104820 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:34:16.105081 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 23:34:16.105386 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 23:34:16.105554 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 23:34:16.106664 28299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:34:16.224603 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 28086
I20260810 23:34:16.248950 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 28315
Aug 10 23:34:16 dist-test-slave-04bb krb5kdc[26950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:34:16.265420 28578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:34:16.249744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.9.48.195:35031)
0810 23:34:16.249931 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:34:16.249936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:34:16.249972 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:34:16.250136 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:34:16.250140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:34:16.250210 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:34:16.250292 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:34:16.251985 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:16.252777 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:16.254334 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:34:16.254719 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:34:16.254825 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:34:16.259694 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:34:16.259709 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:34:16.259725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:34:16.259763 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:34:16.262771 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:16.263492 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:16.263500 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:16.263502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:16.263631 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:34:16.263975 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:34:16.263978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:34:16.263980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:34:16.264337 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:34:16.264635 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0810 23:34:16.264994 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 23:34:16.265158 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 23:34:16.266073 28534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:34:16.355583 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 28315
I20260810 23:34:16.384714 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 27678
I20260810 23:34:16.385959 27777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:34:16.500640 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 27678
I20260810 23:34:16.537139 9411 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 23:34:16.883903 9411 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 23:34:16.884254 9411 mini_ranger.cc:66] Stopping Ranger...
I20260810 23:34:17.239666 9411 mini_ranger.cc:68] Stopped Ranger
2026-08-10 23:34:17.239 UTC [27005] LOG: received smart shutdown request
2026-08-10 23:34:17.244 UTC [27005] LOG: background worker "logical replication launcher" (PID 27021) exited with exit code 1
2026-08-10 23:34:17.244 UTC [27009] LOG: shutting down
2026-08-10 23:34:17.424 UTC [27005] LOG: database system is shut down
2026-08-10T23:34:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52240 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:34:17 dist-test-slave-04bb krb5kdc[28652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:34:17 dist-test-slave-04bb krb5kdc[28652](info): set up 2 sockets
Aug 10 23:34:17 dist-test-slave-04bb krb5kdc[28652](info): commencing operation
krb5kdc: starting...
W20260810 23:34:18.922986 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.298s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:34:19 dist-test-slave-04bb krb5kdc[28652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:34:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:34:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.48.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.48.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/rangeradmin_127.9.48.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/rangeradmin_127.9.48.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.48.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.48.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/rangerlookup_127.9.48.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/rangerlookup_127.9.48.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.48.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.48.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/HTTP_127.9.48.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/HTTP_127.9.48.212@KRBTEST.COM.keytab.
I20260810 23:34:19.325001 9411 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskahRJgu/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/postgres -l logfile start
2026-08-10 23:34:21.948 UTC [28701] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 23:34:21.948 UTC [28701] LOG: listening on IPv4 address "127.9.48.212", port 59287
2026-08-10 23:34:21.952 UTC [28701] LOG: listening on Unix socket "/tmp/.s.PGSQL.59287"
2026-08-10 23:34:21.976 UTC [28707] LOG: database system was shut down at 2026-08-10 23:34:20 UTC
2026-08-10 23:34:21.980 UTC [28701] LOG: database system is ready to accept connections
I20260810 23:34:23.182217 9411 mini_postgres.cc:96] Postgres bound to 59287
2026-08-10 23:34:23.207 UTC [28719] FATAL: database "slave" does not exist
127.9.48.212:59287 - accepting connections
I20260810 23:34:23.208820 9411 mini_ranger.cc:161] Starting Ranger...
I20260810 23:34:23.242344 9411 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 23:34:23.736282 9411 mini_ranger.cc:87] Created ranger Postgres database
I20260810 23:34:23.736420 9411 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-admin
2026-08-10 23:34:24,035 [I] DB FLAVOR :POSTGRES
2026-08-10 23:34:24,036 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:34:24,036 [I] Checking connection..
2026-08-10 23:34:24,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 23:34:24,292 [I] Checking connection passed.
2026-08-10 23:34:24,292 [I] --------- Verifying version history table ---------
2026-08-10 23:34:24,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:34:24,560 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:34:24,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:34:24,836 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:34:24,836 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 23:34:24,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 23:34:25.121 UTC [28826] WARNING: there is no transaction in progress
2026-08-10 23:34:25,128 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 23:34:25,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:34:25,386 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 23:34:25,387 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:34:25,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 23:34:25,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 23:34:25,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-04bb.c.gcp-upstream.internal','N') ;"
2026-08-10 23:34:26,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:34:26,448 [I] Table x_portal_user does not exist in database ranger
2026-08-10 23:34:26,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:34:26,707 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 23:34:26,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:34:26,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:34:27,188 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 23:34:27,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 23:34:28.175 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:28.192 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:28.208 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:28.223 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:28.239 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:28.259 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:28.323 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:28.330 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:28.345 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:28.362 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:28.375 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:28.393 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:28.405 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:28.411 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:28.418 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:28.425 UTC [28992] WARNING: there is no transaction in progress
2026-08-10 23:34:29,086 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 23:34:29,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 23:34:29,352 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 23:34:29,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:34:29,619 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 23:34:29,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:34:29,880 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 23:34:29,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:34:30,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:34:30,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-04bb.c.gcp-upstream.internal';"
2026-08-10 23:34:30,666 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 23:34:30,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 23:34:30,930 [I] DB_PATCHES have already been applied
I20260810 23:34:30.936331 9411 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-admin:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskahRJgu/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 23:34:30.936472 9411 mini_ranger.cc:193] Using host: 127.9.48.212
I20260810 23:34:30.958271 9411 mini_ranger.cc:227] Ranger admin URL: http://127.9.48.212:45289
Aug 10, 2026 11:34:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:34:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:34:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 11:34:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 11:34:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 11:34:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 11:34:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 23:34:32.384403 9411 mini_ranger.cc:160] Time spent starting Ranger: real 9.176s user 0.004s sys 0.061s
Aug 10, 2026 11:34:53 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 11:34:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 10, 2026 11:34:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 11:34:53 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 11:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 11:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 11:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 11:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 11:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 11:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 11:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 11:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 11:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 11:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 11:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 11:34:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 11:34:53 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 11:34:54 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 23:34:54.738940 9411 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.48.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.48.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/rangerkms_127.9.48.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.48.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/rangerkms_127.9.48.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.48.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.48.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.48.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/HTTP_127.9.48.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.48.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/HTTP_127.9.48.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 23:34:54 dist-test-slave-04bb krb5kdc[28652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404894, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 23:34:54.952401 9411 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 23:34:54.980013 9411 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 23:34:55.885437 9411 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 23:34:55.885572 9411 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-kms
I20260810 23:34:55.885625 9411 mini_ranger_kms.cc:227] Using postgres at 127.9.48.212:59287
2026-08-10 23:34:56,295 [I] DB FLAVOR :POSTGRES
2026-08-10 23:34:56,295 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:34:56,295 [I] Checking connection
2026-08-10 23:34:56,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 23:34:56,543 [I] connection success
2026-08-10 23:34:56,543 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 23:34:56,543 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 23:34:56,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 23:34:56,779 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 23:34:56,779 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:34:56,779 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 23:34:56,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.48.212:59287/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 23:34:57,048 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 23:34:57.554075 9411 mini_ranger_kms.cc:310] Created kms service
I20260810 23:34:57.753237 9411 mini_ranger_kms.cc:326] Created kudu user
I20260810 23:34:57.800165 9411 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 23:34:57 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404894, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:34:58.254343 9411 mini_ranger_kms.cc:384] Added ranger policy
I20260810 23:34:58.254714 9411 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-kms:/tmp/dist-test-taskahRJgu/build/asan/bin/postgresql.jar:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskahRJgu/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 23:34:58.254786 9411 mini_ranger_kms.cc:240] Using host: 127.9.48.212
I20260810 23:34:58.284775 9411 mini_ranger_kms.cc:276] Ranger KMS PID: 29292
I20260810 23:34:58.284940 9411 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.48.212:48073
Aug 10, 2026 11:34:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:34:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:34:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 11:34:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 11:34:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 11:34:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 23:35:00 dist-test-slave-04bb krb5kdc[28652](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786404900, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.48.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 23:35:00 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404900, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.48.212@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:35:01.178150 9411 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.226s user 0.095s sys 0.353s
I20260810 23:35:01.178413 9411 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 23:35:01.178596 9411 mini_ranger_kms.cc:401] 127.9.48.212:48073/kms/v1/keys
Aug 10 23:35:01 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404900, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.48.212@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
Aug 10 23:35:03 dist-test-slave-04bb krb5kdc[28652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:35:03.572722 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:33333
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:33333
--ranger_config_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.48.212:48073/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:35:03.917414 29373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:03.917744 29373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:03.917901 29373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:03.925391 29373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:35:03.925546 29373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:03.925654 29373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:03.925757 29373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:35:03.925839 29373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:35:03.935972 29373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.48.212:48073/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:33333
--ranger_config_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:33333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:03.937414 29373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:03.938858 29373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:03.950937 29380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:03.951030 29385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:03.950997 29383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:03.952342 29373 server_base.cc:1034] running on GCE node
I20260810 23:35:03.953158 29373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:03.954608 29373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:03.955852 29373 hybrid_clock.cc:648] HybridClock initialized: now 1786404903955817 us; error 46 us; skew 500 ppm
I20260810 23:35:03.956223 29373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:03 dist-test-slave-04bb krb5kdc[28652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404903, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:03.962963 29373 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:35:03.964475 29373 webserver.cc:460] Webserver started at http://127.9.48.254:38329/ using document root <none> and password file <none>
I20260810 23:35:03.965162 29373 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:03.965265 29373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:03.965529 29373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:35:03 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404903, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:35:04.024399 29373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "6e9696f042f343b9b16a23b5dab777de"
format_stamp: "Formatted at 2026-08-10 23:35:04 on dist-test-slave-04bb"
server_key: "6f3d0b009812a6b2c0cccdda55f2b471"
server_key_iv: "a0df7f281c057d5058e1cf2d1abd0702"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:04.025390 29373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "6e9696f042f343b9b16a23b5dab777de"
format_stamp: "Formatted at 2026-08-10 23:35:04 on dist-test-slave-04bb"
server_key: "6f3d0b009812a6b2c0cccdda55f2b471"
server_key_iv: "a0df7f281c057d5058e1cf2d1abd0702"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:04.031091 29373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 23:35:04.046990 29395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:04.048722 29373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 23:35:04.048928 29373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "6e9696f042f343b9b16a23b5dab777de"
format_stamp: "Formatted at 2026-08-10 23:35:04 on dist-test-slave-04bb"
server_key: "6f3d0b009812a6b2c0cccdda55f2b471"
server_key_iv: "a0df7f281c057d5058e1cf2d1abd0702"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:04.049149 29373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:04.062950 29373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:04.067123 29373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:04.067461 29373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:04.087853 29373 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:33333
I20260810 23:35:04.087873 29475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:33333 every 8 connection(s)
I20260810 23:35:04.089356 29373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:35:04.093905 29477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:04.095194 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 29373
I20260810 23:35:04.095433 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:35:04.096339 9411 external_mini_cluster.cc:1428] Setting key 4517212ab2388c98eae6e7f07fd89e5b
I20260810 23:35:04.106474 29477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de: Bootstrap starting.
I20260810 23:35:04.109755 29477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:04.110750 29477 log.cc:824] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:35:04 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:04.113934 29477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de: No bootstrap required, opened a new log
I20260810 23:35:04.120306 29484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:04.099666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.9.48.254:33333)
0810 23:35:04.100634 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:04.100656 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:04.100717 (+ 61us) server_negotiation.cc:407] Connection header received
0810 23:35:04.102366 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:04.102387 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:04.102762 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:04.103139 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:04.104635 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:04.105922 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:04.107317 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:04.108006 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:04.108218 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:04.112727 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:04.112735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:04.112749 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:04.112784 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:04.115938 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:04.116800 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:04.116818 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:04.116824 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:04.116992 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:04.117398 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:04.117402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:04.117404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:04.117921 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:04.118140 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:04.118714 (+ 574us) server_negotiation.cc:299] Negotiation successful
0810 23:35:04.119293 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":438,"threads_started":1}
I20260810 23:35:04.122447 29477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9696f042f343b9b16a23b5dab777de" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33333 } }
I20260810 23:35:04.122799 29477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:04.122921 29477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e9696f042f343b9b16a23b5dab777de, State: Initialized, Role: FOLLOWER
I20260810 23:35:04.123472 29477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9696f042f343b9b16a23b5dab777de" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33333 } }
I20260810 23:35:04.123698 29477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:35:04.123869 29477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:35:04.124044 29477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:04.126149 29477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9696f042f343b9b16a23b5dab777de" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33333 } }
I20260810 23:35:04.126637 29477 leader_election.cc:302] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e9696f042f343b9b16a23b5dab777de; no voters:
I20260810 23:35:04.127184 29477 leader_election.cc:288] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:35:04.127321 29486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:35:04.128054 29486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 1 LEADER]: Becoming Leader. State: Replica: 6e9696f042f343b9b16a23b5dab777de, State: Running, Role: LEADER
I20260810 23:35:04.128634 29486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9696f042f343b9b16a23b5dab777de" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33333 } }
I20260810 23:35:04.129479 29477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:35:04.133023 29496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e9696f042f343b9b16a23b5dab777de. Latest consensus state: current_term: 1 leader_uuid: "6e9696f042f343b9b16a23b5dab777de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9696f042f343b9b16a23b5dab777de" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33333 } } }
I20260810 23:35:04.133529 29496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:04.134073 29493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e9696f042f343b9b16a23b5dab777de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9696f042f343b9b16a23b5dab777de" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33333 } } }
I20260810 23:35:04.134424 29508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:35:04.134429 29493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:04.139974 29508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:35:04.154456 29508 catalog_manager.cc:1347] Generated new cluster ID: 1c9ef446e54c4807909fd2531f44e44a
I20260810 23:35:04.154593 29508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:35:04.155742 29477 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 23:35:04.179697 29508 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:35:04.181272 29508 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:35:04.187166 29508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de: Generated new TSK 0
I20260810 23:35:04.188001 29508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:35:04.952901 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.48.212:48073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33333
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:35:05 dist-test-slave-04bb krb5kdc[28652](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786404905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 23:35:05.359545 29561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:05.359817 29561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:05.359969 29561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:05.367589 29561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:05.367686 29561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:05.367856 29561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:35:05.379408 29561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.48.212:48073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:05.381110 29561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:05.382958 29561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:05.402130 29576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:05.403771 29577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:05.410701 29579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:05.411854 29561 server_base.cc:1034] running on GCE node
I20260810 23:35:05.412987 29561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:05.417372 29561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:05.418715 29561 hybrid_clock.cc:648] HybridClock initialized: now 1786404905418596 us; error 104 us; skew 500 ppm
I20260810 23:35:05.419117 29561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:05 dist-test-slave-04bb krb5kdc[28652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:05.423506 29561 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:35:05.425599 29561 webserver.cc:460] Webserver started at http://127.9.48.193:45107/ using document root <none> and password file <none>
I20260810 23:35:05.426566 29561 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:05.426748 29561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:05.427156 29561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:35:05 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:35:05.447068 29561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "0c1a7d36c79b45aba51f72eb3d1577c0"
format_stamp: "Formatted at 2026-08-10 23:35:05 on dist-test-slave-04bb"
server_key: "dceb1e436258dd4d47db5e735ab3a050"
server_key_iv: "216e6a155a9fbab7323c36f5bf0d33ec"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:05.448268 29561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "0c1a7d36c79b45aba51f72eb3d1577c0"
format_stamp: "Formatted at 2026-08-10 23:35:05 on dist-test-slave-04bb"
server_key: "dceb1e436258dd4d47db5e735ab3a050"
server_key_iv: "216e6a155a9fbab7323c36f5bf0d33ec"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:05.456604 29561 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260810 23:35:05.484714 29588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:05.491151 29561 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.002s sys 0.001s
I20260810 23:35:05.491362 29561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "0c1a7d36c79b45aba51f72eb3d1577c0"
format_stamp: "Formatted at 2026-08-10 23:35:05 on dist-test-slave-04bb"
server_key: "dceb1e436258dd4d47db5e735ab3a050"
server_key_iv: "216e6a155a9fbab7323c36f5bf0d33ec"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:05.491691 29561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:05.508466 29561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:05.512787 29561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:05.513100 29561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:05.514151 29561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:05.515712 29561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:05.515825 29561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:05.515986 29561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:05.516065 29561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:05.566833 29561 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:33769
I20260810 23:35:05.566840 29763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:33769 every 8 connection(s)
I20260810 23:35:05.568614 29561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:35:05.576977 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 29561
I20260810 23:35:05.577262 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:35:05.578285 9411 external_mini_cluster.cc:1428] Setting key f6c134694872f7676df1745970998a7a
Aug 10 23:35:05 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:05.617218 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:05.579523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:43889 (local address 127.9.48.254:33333)
0810 23:35:05.580005 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:05.580012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:05.592597 (+ 12585us) server_negotiation.cc:407] Connection header received
0810 23:35:05.596612 (+ 4015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:05.596621 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:05.596743 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:05.596839 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:05.598997 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:05.600292 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:05.603032 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:05.603817 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:05.603935 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:05.608932 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:05.608940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:05.608948 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:05.608995 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:05.613101 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:05.613933 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:05.613941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:05.613948 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:05.614113 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:05.614771 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:05.614779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:05.614785 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:05.615220 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:05.615393 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:05.616468 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0810 23:35:05.616746 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":1271,"threads_started":1}
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
I20260810 23:35:05.619547 29769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:05.575903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33333 (local address 127.9.48.193:43889)
0810 23:35:05.592375 (+ 16472us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:05.592441 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:05.596200 (+ 3759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:05.597445 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:05.597455 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:05.597914 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:05.598793 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:05.598821 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:05.600502 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:05.600510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:05.602867 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:05.602885 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:05.604301 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:05.604310 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:05.604679 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:05.605553 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:05.605588 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:05.608688 (+ 3100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:05.613311 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:05.613328 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:05.613346 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:05.613792 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:05.614299 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:05.614307 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:05.614313 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:05.614615 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:05.615530 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:05.615540 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:05.615989 (+ 449us) client_negotiation.cc:769] Sending connection context
0810 23:35:05.617522 (+ 1533us) client_negotiation.cc:240] Negotiation successful
0810 23:35:05.618149 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16064,"thread_start_us":16565,"threads_started":1}
I20260810 23:35:05.621390 29765 heartbeater.cc:347] Connected to a master server at 127.9.48.254:33333
I20260810 23:35:05.621800 29765 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:05.622534 29765 heartbeater.cc:511] Master 127.9.48.254:33333 requested a full tablet report, sending...
I20260810 23:35:05.625308 29419 ts_manager.cc:194] Registered new tserver with Master: 0c1a7d36c79b45aba51f72eb3d1577c0 (127.9.48.193:33769)
I20260810 23:35:05.627314 29419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:43889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:35:05.668073 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.48.212:48073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33333
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:35:05 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
Aug 10 23:35:06 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404905, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
W20260810 23:35:06.083956 29778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:06.084425 29778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:06.084645 29778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:06.097277 29778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:06.097455 29778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:06.097683 29778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:35:06.113834 29778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.48.212:48073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:06.115809 29778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:06.117893 29778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:06.138854 29788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:06.140285 29789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:06.176883 29778 server_base.cc:1034] running on GCE node
W20260810 23:35:06.178287 29795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:06.179343 29778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:06.200906 29778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:06.202260 29778 hybrid_clock.cc:648] HybridClock initialized: now 1786404906202188 us; error 84 us; skew 500 ppm
I20260810 23:35:06.202705 29778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:06 dist-test-slave-04bb krb5kdc[28652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404906, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:06.209300 29778 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:35:06.222090 29778 webserver.cc:460] Webserver started at http://127.9.48.194:45623/ using document root <none> and password file <none>
I20260810 23:35:06.222932 29778 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:06.223098 29778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:06.223469 29778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:35:06 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404906, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:35:06.251538 29778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "3d2c65445d394cc198402508da57aa5c"
format_stamp: "Formatted at 2026-08-10 23:35:06 on dist-test-slave-04bb"
server_key: "1d76ea0fda5d9603b8dffe63881fba51"
server_key_iv: "128edb9ef1a2f5ddeaa18d0d0e2ae8eb"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:06.252942 29778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "3d2c65445d394cc198402508da57aa5c"
format_stamp: "Formatted at 2026-08-10 23:35:06 on dist-test-slave-04bb"
server_key: "1d76ea0fda5d9603b8dffe63881fba51"
server_key_iv: "128edb9ef1a2f5ddeaa18d0d0e2ae8eb"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:06.267414 29778 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.008s sys 0.004s
I20260810 23:35:06.301239 29807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:06.310118 29778 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.001s sys 0.006s
I20260810 23:35:06.310381 29778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "3d2c65445d394cc198402508da57aa5c"
format_stamp: "Formatted at 2026-08-10 23:35:06 on dist-test-slave-04bb"
server_key: "1d76ea0fda5d9603b8dffe63881fba51"
server_key_iv: "128edb9ef1a2f5ddeaa18d0d0e2ae8eb"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:06.310662 29778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:06.340376 29778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:06.346493 29778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:06.346869 29778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:06.348165 29778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:06.350131 29778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:06.350306 29778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 23:35:06.350449 29778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:06.350521 29778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:06.412848 29778 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:34851
I20260810 23:35:06.412871 30012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:34851 every 8 connection(s)
I20260810 23:35:06.414215 29778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:35:06.422086 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 29778
I20260810 23:35:06.422276 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:35:06.422843 9411 external_mini_cluster.cc:1428] Setting key 375cc025f077bc2992f5d449a235907b
Aug 10 23:35:06 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404906, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:06.438690 30017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:06.417260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:41271 (local address 127.9.48.254:33333)
0810 23:35:06.417801 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:06.417806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:06.418907 (+ 1101us) server_negotiation.cc:407] Connection header received
0810 23:35:06.420830 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:06.420835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:06.420941 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:06.421063 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:06.422647 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:06.423632 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:06.425958 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:06.426581 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:06.426715 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:06.431366 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:06.431372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:06.431377 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:06.431415 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:06.434911 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:06.435910 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:06.435916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:06.435920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:06.436070 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:06.436648 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:06.436653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:06.436656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:06.436988 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:06.437142 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:06.438051 (+ 909us) server_negotiation.cc:299] Negotiation successful
0810 23:35:06.438291 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":277,"threads_started":1}
I20260810 23:35:06.439837 30018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:06.417883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33333 (local address 127.9.48.194:41271)
0810 23:35:06.418733 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:06.418780 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:06.420481 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:06.421317 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:06.421325 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:06.421776 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:06.422470 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:06.422491 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:06.423779 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:06.423785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:06.425792 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:06.425809 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:06.426842 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:06.426852 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:06.427179 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:06.428064 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:06.428098 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:06.431124 (+ 3026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:06.435068 (+ 3944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:06.435080 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:06.435099 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:06.435767 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:06.436190 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:06.436197 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:06.436203 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:06.436508 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:06.437145 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:06.437155 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:06.437616 (+ 461us) client_negotiation.cc:769] Sending connection context
0810 23:35:06.437923 (+ 307us) client_negotiation.cc:240] Negotiation successful
0810 23:35:06.438458 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":377,"threads_started":1}
I20260810 23:35:06.441421 30015 heartbeater.cc:347] Connected to a master server at 127.9.48.254:33333
I20260810 23:35:06.441747 30015 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:06.442461 30015 heartbeater.cc:511] Master 127.9.48.254:33333 requested a full tablet report, sending...
I20260810 23:35:06.443933 29419 ts_manager.cc:194] Registered new tserver with Master: 3d2c65445d394cc198402508da57aa5c (127.9.48.194:34851)
I20260810 23:35:06.444832 29419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:41271
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:35:06.496049 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.48.212:48073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33333
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:35:06.631052 29765 heartbeater.cc:503] Master 127.9.48.254:33333 was elected leader, sending a full tablet report...
W20260810 23:35:06.883198 30023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:06.883528 30023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:06.883703 30023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:06.891119 30023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:06.891280 30023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:06.891455 30023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:35:06.901857 30023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.48.212:48073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:06.903342 30023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:06.904819 30023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:06.919929 30030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:06.920061 30036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:06.920056 30029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:06.921990 30023 server_base.cc:1034] running on GCE node
I20260810 23:35:06.922729 30023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:06.923780 30023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:06.925015 30023 hybrid_clock.cc:648] HybridClock initialized: now 1786404906924948 us; error 60 us; skew 500 ppm
I20260810 23:35:06.925369 30023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:06 dist-test-slave-04bb krb5kdc[28652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404906, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:06.929256 30023 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:35:06.931013 30023 webserver.cc:460] Webserver started at http://127.9.48.195:37723/ using document root <none> and password file <none>
I20260810 23:35:06.931751 30023 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:06.931901 30023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:06.932228 30023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:35:06 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404906, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:35:06.951234 30023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3"
format_stamp: "Formatted at 2026-08-10 23:35:06 on dist-test-slave-04bb"
server_key: "bbfe77762695fc17f2dd4656e626896e"
server_key_iv: "d508dfe746f6233397cf2559d4d68fba"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:06.952378 30023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3"
format_stamp: "Formatted at 2026-08-10 23:35:06 on dist-test-slave-04bb"
server_key: "bbfe77762695fc17f2dd4656e626896e"
server_key_iv: "d508dfe746f6233397cf2559d4d68fba"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:06.959965 30023 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260810 23:35:06.975816 30044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:06.977419 30023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:35:06.977622 30023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3"
format_stamp: "Formatted at 2026-08-10 23:35:06 on dist-test-slave-04bb"
server_key: "bbfe77762695fc17f2dd4656e626896e"
server_key_iv: "d508dfe746f6233397cf2559d4d68fba"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:06.977833 30023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:06.994477 30023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:06.998585 30023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:06.998884 30023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:06.999866 30023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:07.001360 30023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:07.001439 30023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:07.001605 30023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:07.001657 30023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:07.037427 30023 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:46575
I20260810 23:35:07.037515 30236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:46575 every 8 connection(s)
I20260810 23:35:07.038758 30023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:35:07.039479 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 30023
I20260810 23:35:07.039697 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:35:07.040369 9411 external_mini_cluster.cc:1428] Setting key 91d45d5c0cbfd63dd8f76c7ccc0ca344
Aug 10 23:35:07 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404906, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:07.059571 30241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:07.041616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:42211 (local address 127.9.48.254:33333)
0810 23:35:07.042298 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:07.042304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:07.043275 (+ 971us) server_negotiation.cc:407] Connection header received
0810 23:35:07.045358 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:07.045363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:07.045460 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:07.045587 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:07.047017 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:07.047734 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:07.049403 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:07.049853 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:07.049984 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:07.053722 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:07.053727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:07.053731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:07.053765 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:07.056788 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:07.057516 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:07.057520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:07.057523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:07.057647 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:07.058058 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:07.058064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:07.058069 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:07.058412 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:07.058526 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:07.059160 (+ 634us) server_negotiation.cc:299] Negotiation successful
0810 23:35:07.059318 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":413,"threads_started":1}
I20260810 23:35:07.060506 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:07.042319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33333 (local address 127.9.48.195:42211)
0810 23:35:07.043124 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:07.043163 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:07.045101 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:07.045839 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:07.045847 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:07.046274 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:07.046848 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:07.046865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:07.047913 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:07.047919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:07.049243 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:07.049263 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:07.050103 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:07.050113 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:07.050357 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:07.051207 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:07.051244 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:07.053529 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:07.056918 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:07.056926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:07.056937 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:07.057382 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:07.057744 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:07.057747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:07.057750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:07.057925 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:07.058530 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:07.058536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:07.058842 (+ 306us) client_negotiation.cc:769] Sending connection context
0810 23:35:07.059065 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 23:35:07.059453 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":351,"threads_started":1}
I20260810 23:35:07.061863 30237 heartbeater.cc:347] Connected to a master server at 127.9.48.254:33333
I20260810 23:35:07.062175 30237 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:07.062906 30237 heartbeater.cc:511] Master 127.9.48.254:33333 requested a full tablet report, sending...
I20260810 23:35:07.064446 29419 ts_manager.cc:194] Registered new tserver with Master: 7b6a4b5cbd924abdb54b04f7fdfe61e3 (127.9.48.195:46575)
I20260810 23:35:07.065392 29419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:42211
I20260810 23:35:07.068517 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:35:07.070232 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 29561
Aug 10 23:35:07 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:35:07.087741 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:07.071166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40570 (local address 127.9.48.193:33769)
0810 23:35:07.071806 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:07.071812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:07.071836 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:35:07.071892 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:07.071898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:07.072072 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:07.072200 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:07.073761 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:07.074757 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:07.076220 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:07.076894 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:07.077037 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:07.081483 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:07.081495 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:07.081508 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:07.081543 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:07.084660 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:07.085417 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:07.085425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:07.085427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:07.085581 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:07.086011 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:07.086019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:07.086023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:07.086359 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:07.086714 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:07.087054 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 23:35:07.087369 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":350,"threads_started":1}
I20260810 23:35:07.089344 29759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:07.174901 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 29561
I20260810 23:35:07.203935 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 29778
Aug 10 23:35:07 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:35:07.221189 30249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:07.204909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.9.48.194:34851)
0810 23:35:07.205502 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:07.205509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:07.205536 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:35:07.205597 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:07.205603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:07.205835 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:07.206012 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:07.207280 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:07.208315 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:07.209785 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:07.210357 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:07.210507 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:07.215052 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:07.215063 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:07.215077 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:07.215119 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:07.218023 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:07.218768 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:07.218776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:07.218778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:07.218936 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:07.219326 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:07.219331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:07.219340 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:07.219697 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:07.220019 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:07.220353 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 23:35:07.220702 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":312,"threads_started":1}
I20260810 23:35:07.222503 30008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:07.306483 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 29778
I20260810 23:35:07.335393 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 30023
Aug 10 23:35:07 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:35:07.352005 30253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:07.336366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.9.48.195:46575)
0810 23:35:07.337046 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:07.337054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:07.337083 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:35:07.337144 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:07.337150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:07.337370 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:07.337551 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:07.338924 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:07.339964 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:07.341431 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:07.342005 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:07.342154 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:07.346457 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:07.346468 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:07.346483 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:07.346520 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:07.349503 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:07.350222 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:07.350229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:07.350231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:07.350381 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:07.350719 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:07.350722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:07.350724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:07.351046 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:07.351297 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:07.351505 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 23:35:07.351722 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":366,"threads_started":1}
I20260810 23:35:07.353477 30230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:07.437041 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 30023
I20260810 23:35:07.461881 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 29373
I20260810 23:35:07.463270 29459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:07.570114 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 29373
I20260810 23:35:07.607285 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:33333
--webserver_interface=127.9.48.254
--webserver_port=38329
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:33333
--ranger_config_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.48.212:48073/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.48.212:48073/kms with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:35:07.938969 30261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:07.939254 30261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:07.939345 30261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:07.946441 30261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:35:07.946547 30261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:07.946594 30261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:07.946650 30261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:35:07.946692 30261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:35:07.956604 30261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.48.212:48073/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:33333
--ranger_config_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:33333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=38329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:07.957806 30261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:07.959092 30261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:07.970827 30271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:07.970880 30272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:07.971048 30275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:07.971849 30261 server_base.cc:1034] running on GCE node
I20260810 23:35:07.972683 30261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:07.974150 30261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:07.975389 30261 hybrid_clock.cc:648] HybridClock initialized: now 1786404907975324 us; error 55 us; skew 500 ppm
I20260810 23:35:07.975759 30261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:07 dist-test-slave-04bb krb5kdc[28652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404907, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:07.978981 30261 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:35:07.980487 30261 webserver.cc:460] Webserver started at http://127.9.48.254:38329/ using document root <none> and password file <none>
I20260810 23:35:07.981199 30261 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:07.981351 30261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:35:08 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404907, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:35:08.076027 30261 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:35:08.079910 30290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:08.081683 30261 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260810 23:35:08.081910 30261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "6e9696f042f343b9b16a23b5dab777de"
format_stamp: "Formatted at 2026-08-10 23:35:04 on dist-test-slave-04bb"
server_key: "6f3d0b009812a6b2c0cccdda55f2b471"
server_key_iv: "a0df7f281c057d5058e1cf2d1abd0702"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:08.082895 30261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:08.100217 30261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:08.104085 30261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:08.104380 30261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:08.123855 30261 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:33333
I20260810 23:35:08.123883 30382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:33333 every 8 connection(s)
I20260810 23:35:08.125504 30261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:35:08.126142 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 30261
I20260810 23:35:08.127308 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.48.212:48073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:33769
--local_ip_for_outbound_sockets=127.9.48.193
--tserver_master_addrs=127.9.48.254:33333
--webserver_port=45107
--webserver_interface=127.9.48.193
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.48.212:48073/kms with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:35:08.132870 30383 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:35:08.136312 30383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de: Bootstrap starting.
I20260810 23:35:08.162853 30383 log.cc:824] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:08.174486 30383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:35:08.175130 30383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de: Bootstrap complete.
I20260810 23:35:08.188469 30383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9696f042f343b9b16a23b5dab777de" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33333 } }
I20260810 23:35:08.189723 30383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e9696f042f343b9b16a23b5dab777de, State: Initialized, Role: FOLLOWER
I20260810 23:35:08.190517 30383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9696f042f343b9b16a23b5dab777de" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33333 } }
I20260810 23:35:08.190819 30383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:35:08.190984 30383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:35:08.191179 30383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:35:08.197186 30383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9696f042f343b9b16a23b5dab777de" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33333 } }
I20260810 23:35:08.197793 30383 leader_election.cc:302] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e9696f042f343b9b16a23b5dab777de; no voters:
I20260810 23:35:08.198460 30383 leader_election.cc:288] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:35:08.198645 30388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:35:08.200037 30388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [term 2 LEADER]: Becoming Leader. State: Replica: 6e9696f042f343b9b16a23b5dab777de, State: Running, Role: LEADER
I20260810 23:35:08.200664 30383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:35:08.200690 30388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9696f042f343b9b16a23b5dab777de" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33333 } }
I20260810 23:35:08.203259 30390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e9696f042f343b9b16a23b5dab777de. Latest consensus state: current_term: 2 leader_uuid: "6e9696f042f343b9b16a23b5dab777de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9696f042f343b9b16a23b5dab777de" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33333 } } }
I20260810 23:35:08.203630 30390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:08.204174 30389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6e9696f042f343b9b16a23b5dab777de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9696f042f343b9b16a23b5dab777de" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33333 } } }
I20260810 23:35:08.204517 30389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:08.214931 30399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:35:08.220939 30399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:35:08.222414 30399 catalog_manager.cc:1259] Loaded cluster ID: 1c9ef446e54c4807909fd2531f44e44a
I20260810 23:35:08.222584 30399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:35:08.226964 30399 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:35:08.228701 30399 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de: Loaded TSK: 0
I20260810 23:35:08.230345 30399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:35:08.233376 30383 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260810 23:35:08.628290 30385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:08.628863 30385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:08.629106 30385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:08.644309 30385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:08.644567 30385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:08.644830 30385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:35:08.666667 30385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.48.212:48073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:33769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=45107
--tserver_master_addrs=127.9.48.254:33333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:08.668869 30385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:08.670876 30385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:08.702720 30439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:08.705039 30440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:08.706557 30443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:08.709527 30385 server_base.cc:1034] running on GCE node
I20260810 23:35:08.710319 30385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:08.711578 30385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:08.712826 30385 hybrid_clock.cc:648] HybridClock initialized: now 1786404908712721 us; error 59 us; skew 500 ppm
I20260810 23:35:08.713236 30385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:08 dist-test-slave-04bb krb5kdc[28652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404908, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:08.718093 30385 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:35:08.720060 30385 webserver.cc:460] Webserver started at http://127.9.48.193:45107/ using document root <none> and password file <none>
I20260810 23:35:08.720920 30385 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:08.721093 30385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:35:08 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404908, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:35:08.765738 30385 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.004s sys 0.000s
I20260810 23:35:08.779300 30463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:08.781651 30385 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.005s sys 0.008s
I20260810 23:35:08.781867 30385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "0c1a7d36c79b45aba51f72eb3d1577c0"
format_stamp: "Formatted at 2026-08-10 23:35:05 on dist-test-slave-04bb"
server_key: "dceb1e436258dd4d47db5e735ab3a050"
server_key_iv: "216e6a155a9fbab7323c36f5bf0d33ec"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:08.783376 30385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:08.807264 30385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:08.811960 30385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:08.812268 30385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:08.813608 30385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:08.815358 30385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:08.815474 30385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:08.815577 30385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:08.815651 30385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:08.881841 30385 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:33769
I20260810 23:35:08.881883 30655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:33769 every 8 connection(s)
I20260810 23:35:08.883464 30385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:35:08.892189 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 30385
I20260810 23:35:08.893221 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.48.212:48073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:34851
--local_ip_for_outbound_sockets=127.9.48.194
--tserver_master_addrs=127.9.48.254:33333
--webserver_port=45623
--webserver_interface=127.9.48.194
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.48.212:48073/kms with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:35:08 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404908, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:08.941757 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:08.889080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33333 (local address 127.9.48.193:46479)
0810 23:35:08.909915 (+ 20835us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:08.909991 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:08.912792 (+ 2801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:08.916894 (+ 4102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:08.916906 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:08.917640 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:08.918532 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:08.918566 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:08.921653 (+ 3087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:08.921659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:08.923802 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:08.923818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:08.930128 (+ 6310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:08.930135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:08.930336 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:08.931101 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:08.931126 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:08.933296 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:08.937075 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:08.937084 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:08.937100 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:08.937543 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:08.938030 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:08.938035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:08.938039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:08.938231 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:08.939037 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:08.939046 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:08.939390 (+ 344us) client_negotiation.cc:769] Sending connection context
0810 23:35:08.939660 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 23:35:08.940250 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20489,"thread_start_us":20294,"threads_started":1}
I20260810 23:35:08.941848 30663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:08.890240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:46479 (local address 127.9.48.254:33333)
0810 23:35:08.909189 (+ 18949us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:08.909231 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:08.913234 (+ 4003us) server_negotiation.cc:407] Connection header received
0810 23:35:08.913724 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:08.913744 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:08.914244 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:08.914663 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:08.919405 (+ 4742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:08.921422 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:08.928414 (+ 6992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:08.929558 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:08.929764 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:08.933577 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:08.933587 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:08.933605 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:08.933650 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:08.936925 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:08.937711 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:08.937719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:08.937726 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:08.937900 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:08.938367 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:08.938379 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:08.938382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:08.938884 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:08.939094 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:08.939937 (+ 843us) server_negotiation.cc:299] Negotiation successful
0810 23:35:08.940341 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18590,"thread_start_us":18385,"threads_started":1}
I20260810 23:35:08.945457 30658 heartbeater.cc:347] Connected to a master server at 127.9.48.254:33333
I20260810 23:35:08.945820 30658 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:08.946576 30658 heartbeater.cc:511] Master 127.9.48.254:33333 requested a full tablet report, sending...
W20260810 23:35:08.949949 30658 heartbeater.cc:644] Failed to heartbeat to 127.9.48.254:33333 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:35:08.950167 30658 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:08.950448 30658 heartbeater.cc:511] Master 127.9.48.254:33333 requested a full tablet report, sending...
I20260810 23:35:08.951794 30658 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:08.952122 30658 heartbeater.cc:511] Master 127.9.48.254:33333 requested a full tablet report, sending...
W20260810 23:35:08.953318 30658 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:35:09.331664 30664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:09.331972 30664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:09.332144 30664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:09.339515 30664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:09.339614 30664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:09.339789 30664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:35:09.353004 30664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.48.212:48073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:34851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=45623
--tserver_master_addrs=127.9.48.254:33333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:09.354628 30664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:09.356367 30664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:09.372830 30685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:09.372856 30686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:09.374645 30664 server_base.cc:1034] running on GCE node
W20260810 23:35:09.375036 30688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:09.375859 30664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:09.377130 30664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:09.378418 30664 hybrid_clock.cc:648] HybridClock initialized: now 1786404909378354 us; error 51 us; skew 500 ppm
I20260810 23:35:09.378877 30664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:09 dist-test-slave-04bb krb5kdc[28652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404909, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:09.389302 30664 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:35:09.433017 30664 webserver.cc:460] Webserver started at http://127.9.48.194:45623/ using document root <none> and password file <none>
I20260810 23:35:09.434216 30664 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:09.434473 30664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:35:09 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404909, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:35:09.474402 30664 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.004s sys 0.008s
I20260810 23:35:09.488230 30708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:09.491024 30664 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.009s sys 0.005s
I20260810 23:35:09.491334 30664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "3d2c65445d394cc198402508da57aa5c"
format_stamp: "Formatted at 2026-08-10 23:35:06 on dist-test-slave-04bb"
server_key: "1d76ea0fda5d9603b8dffe63881fba51"
server_key_iv: "128edb9ef1a2f5ddeaa18d0d0e2ae8eb"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:09.492987 30664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 10 23:35:09 dist-test-slave-04bb krb5kdc[28652](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786404909, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:09.550909 30664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:09.555791 30664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:09.556098 30664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:09.557310 30664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:09.559020 30664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:09.559131 30664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:09.559242 30664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:09.559315 30664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:09.610160 30664 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:34851
I20260810 23:35:09.610213 30910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:34851 every 8 connection(s)
I20260810 23:35:09.612159 30664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:35:09.615347 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 30664
I20260810 23:35:09.616180 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.48.212:48073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:46575
--local_ip_for_outbound_sockets=127.9.48.195
--tserver_master_addrs=127.9.48.254:33333
--webserver_port=37723
--webserver_interface=127.9.48.195
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.48.212:48073/kms with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:35:09 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404909, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:09.670061 30919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:09.616787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:40473 (local address 127.9.48.254:33333)
0810 23:35:09.644873 (+ 28086us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:09.644880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:09.644907 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:35:09.647539 (+ 2632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:09.647546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:09.647636 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:09.647716 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:09.652703 (+ 4987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:09.653878 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:09.656454 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:09.657154 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:09.657265 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:09.661922 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:09.661929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:09.661935 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:09.661978 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:09.665811 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:09.666986 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:09.666993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:09.666997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:09.667170 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:09.667746 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:09.667751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:09.667754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:09.668183 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:09.668323 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:09.669417 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0810 23:35:09.669661 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27953,"thread_start_us":28998,"threads_started":1}
I20260810 23:35:09.672782 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:09.615872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33333 (local address 127.9.48.194:40473)
0810 23:35:09.642878 (+ 27006us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:09.642953 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:09.647180 (+ 4227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:09.648279 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:09.648293 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:09.649064 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:09.650015 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:09.650045 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:09.654136 (+ 4091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:09.654146 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:09.656286 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:09.656302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:09.657658 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:09.657668 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:09.657971 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:09.658821 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:09.658854 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:09.661656 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:09.665998 (+ 4342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:09.666009 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:09.666028 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:09.666836 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:09.667327 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:09.667334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:09.667339 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:09.667603 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:09.668419 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:09.668429 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:09.668927 (+ 498us) client_negotiation.cc:769] Sending connection context
0810 23:35:09.670636 (+ 1709us) client_negotiation.cc:240] Negotiation successful
0810 23:35:09.671243 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26623,"thread_start_us":28921,"threads_started":1}
I20260810 23:35:09.674823 30913 heartbeater.cc:347] Connected to a master server at 127.9.48.254:33333
I20260810 23:35:09.675177 30913 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:09.676137 30913 heartbeater.cc:511] Master 127.9.48.254:33333 requested a full tablet report, sending...
I20260810 23:35:09.678845 30320 ts_manager.cc:194] Registered new tserver with Master: 3d2c65445d394cc198402508da57aa5c (127.9.48.194:34851)
I20260810 23:35:09.681526 30320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:40473
I20260810 23:35:09.953516 30658 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:09.954090 30658 heartbeater.cc:511] Master 127.9.48.254:33333 requested a full tablet report, sending...
I20260810 23:35:09.955587 30320 ts_manager.cc:194] Registered new tserver with Master: 0c1a7d36c79b45aba51f72eb3d1577c0 (127.9.48.193:33769)
I20260810 23:35:09.956991 30320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:46479
Aug 10 23:35:10 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404909, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
W20260810 23:35:10.148054 30920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:10.148658 30920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:10.148830 30920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:10.161278 30920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:10.161434 30920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:10.161605 30920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:35:10.173411 30920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.48.212:48073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:46575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=37723
--tserver_master_addrs=127.9.48.254:33333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:10.174906 30920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:10.176613 30920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:10.198858 30931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:10.198773 30930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:10.203099 30934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:10.203605 30920 server_base.cc:1034] running on GCE node
I20260810 23:35:10.204376 30920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:10.205755 30920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:10.207042 30920 hybrid_clock.cc:648] HybridClock initialized: now 1786404910206970 us; error 48 us; skew 500 ppm
I20260810 23:35:10.207468 30920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:10 dist-test-slave-04bb krb5kdc[28652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404910, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:10.212165 30920 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:35:10.214349 30920 webserver.cc:460] Webserver started at http://127.9.48.195:37723/ using document root <none> and password file <none>
I20260810 23:35:10.215443 30920 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:10.215636 30920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:35:10 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404910, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:35:10.239336 30920 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 23:35:10.242982 30949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:10.244728 30920 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:35:10.244992 30920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3"
format_stamp: "Formatted at 2026-08-10 23:35:06 on dist-test-slave-04bb"
server_key: "bbfe77762695fc17f2dd4656e626896e"
server_key_iv: "d508dfe746f6233397cf2559d4d68fba"
server_key_version: "kuduclusterkey@0"
I20260810 23:35:10.246337 30920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:10.274694 30920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:10.280289 30920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:10.280655 30920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:10.282662 30920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:10.285012 30920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:10.285125 30920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:10.285229 30920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:10.285303 30920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:35:10 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404909, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for HTTP/127.9.48.212@KRBTEST.COM
I20260810 23:35:10.605939 30920 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:46575
I20260810 23:35:10.606022 31146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:46575 every 8 connection(s)
I20260810 23:35:10.608475 30920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:35:10.613184 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 30920
Aug 10 23:35:10 dist-test-slave-04bb krb5kdc[28652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404910, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:10.635443 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:10.614498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.9.48.254:33333)
0810 23:35:10.617823 (+ 3325us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:10.617833 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:10.618782 (+ 949us) server_negotiation.cc:407] Connection header received
0810 23:35:10.618862 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:10.618868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:10.618981 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:10.619093 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:10.621063 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.622203 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.623748 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.624685 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.624829 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:10.628582 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:10.628593 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:10.628601 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:10.628661 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:10.632207 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:10.633121 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:10.633131 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:10.633138 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:10.633339 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:10.633794 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:10.633800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:10.633804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:10.634161 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:10.634500 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:10.634817 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 23:35:10.634999 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":3122,"thread_start_us":923,"threads_started":1}
I20260810 23:35:10.640976 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:10.614005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:47405 (local address 127.9.48.254:33333)
0810 23:35:10.615736 (+ 1731us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:10.615746 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:10.619367 (+ 3621us) server_negotiation.cc:407] Connection header received
0810 23:35:10.621657 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:10.621665 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:10.621759 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:10.621876 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:10.623934 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.625327 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.627212 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.627708 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.627866 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:10.632345 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:10.632354 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:10.632359 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:10.632402 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:10.637504 (+ 5102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:10.638562 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:10.638566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:10.638568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:10.638681 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:10.639219 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:10.639223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:10.639225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:10.639468 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:10.639576 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:10.640478 (+ 902us) server_negotiation.cc:299] Negotiation successful
0810 23:35:10.640715 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":1533,"thread_start_us":3411,"threads_started":1}
I20260810 23:35:10.642498 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:10.613147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33333 (local address 127.9.48.195:47405)
0810 23:35:10.618828 (+ 5681us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:10.618887 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:10.621330 (+ 2443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:10.622182 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:10.622193 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:10.622839 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:10.623703 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:10.623733 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.625496 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.625502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:10.627030 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:10.627052 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.628010 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.628020 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:10.628259 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:10.629014 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:10.629052 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:10.632050 (+ 2998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:10.637664 (+ 5614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:10.637675 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:10.637693 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:10.638401 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:10.638800 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:10.638813 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:10.638817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:10.639076 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:10.639604 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:10.639613 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:10.640081 (+ 468us) client_negotiation.cc:769] Sending connection context
0810 23:35:10.640419 (+ 338us) client_negotiation.cc:240] Negotiation successful
0810 23:35:10.641036 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5350,"thread_start_us":5073,"threads_started":1}
I20260810 23:35:10.643963 31156 heartbeater.cc:347] Connected to a master server at 127.9.48.254:33333
I20260810 23:35:10.644315 31156 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:10.645128 31156 heartbeater.cc:511] Master 127.9.48.254:33333 requested a full tablet report, sending...
I20260810 23:35:10.646590 30320 ts_manager.cc:194] Registered new tserver with Master: 7b6a4b5cbd924abdb54b04f7fdfe61e3 (127.9.48.195:46575)
I20260810 23:35:10.647395 30320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:47405
I20260810 23:35:10.651178 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:35:10.669972 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:10.654769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41708 (local address 127.9.48.254:33333)
0810 23:35:10.654964 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:10.654971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:10.655474 (+ 503us) server_negotiation.cc:407] Connection header received
0810 23:35:10.655588 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:10.655594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:10.655671 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:10.655775 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:10.657348 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.658412 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.659734 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.660379 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.660504 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:10.663302 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:10.663309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:10.663314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:10.663353 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:10.666810 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:10.667563 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:10.667569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:10.667573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:10.667744 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:10.668325 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:10.668331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:10.668335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:10.668721 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:10.668981 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:10.669353 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 23:35:10.669604 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:35:10.675144 30320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:35:10.682869 30320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:35:10.686782 30913 heartbeater.cc:503] Master 127.9.48.254:33333 was elected leader, sending a full tablet report...
I20260810 23:35:10.729013 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:10.698468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53758 (local address 127.9.48.195:46575)
0810 23:35:10.715351 (+ 16883us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:10.715361 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:10.718211 (+ 2850us) server_negotiation.cc:407] Connection header received
0810 23:35:10.718325 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:10.718332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:10.718581 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:10.718770 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:10.719912 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.721498 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.725544 (+ 4046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.727076 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.727220 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:10.727466 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:10.728453 (+ 987us) server_negotiation.cc:299] Negotiation successful
0810 23:35:10.728666 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16694,"thread_start_us":16895,"threads_started":1}
I20260810 23:35:10.729043 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:10.698120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:33769 (local address 127.0.0.1:40584)
0810 23:35:10.717943 (+ 19823us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:10.717966 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:10.718087 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:10.719298 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:10.719310 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:10.719352 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:10.719952 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:10.719968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.722609 (+ 2641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.722621 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:10.725169 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:10.725184 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.727686 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.727691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:10.728024 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:10.728172 (+ 148us) client_negotiation.cc:769] Sending connection context
0810 23:35:10.728303 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 23:35:10.728508 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19692,"mutex_wait_us":122,"thread_start_us":19495,"threads_started":1}
I20260810 23:35:10.729894 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:10.697926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:46575 (local address 127.0.0.1:53758)
0810 23:35:10.717040 (+ 19114us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:10.717084 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:10.717319 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:10.719125 (+ 1806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:10.719133 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:10.719170 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:10.719653 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:10.719668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.721674 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.721679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:10.725384 (+ 3705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:10.725394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.727667 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.727677 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:10.728025 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:10.728170 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 23:35:10.728303 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 23:35:10.729572 (+ 1269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18831,"mutex_wait_us":2380,"thread_start_us":26301,"threads_started":1}
I20260810 23:35:10.730553 31183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:10.698962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40584 (local address 127.9.48.193:33769)
0810 23:35:10.716400 (+ 17438us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:10.716409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:10.718101 (+ 1692us) server_negotiation.cc:407] Connection header received
0810 23:35:10.718258 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:10.718271 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:10.718606 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:10.718991 (+ 385us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:10.720224 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.722395 (+ 2171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.725568 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.727076 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.727221 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:10.727467 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:10.730181 (+ 2714us) server_negotiation.cc:299] Negotiation successful
0810 23:35:10.730324 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17132,"spinlock_wait_cycles":197504,"thread_start_us":17014,"threads_started":1}
I20260810 23:35:10.733633 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:10.708701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:34851 (local address 127.0.0.1:32982)
0810 23:35:10.724541 (+ 15840us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:10.724560 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:10.724774 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:10.725884 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:10.725889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:10.725914 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:10.726243 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:10.726253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.728601 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.728606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:10.730727 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:10.730742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.732817 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.732824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:10.733014 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:10.733048 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:35:10.733156 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 23:35:10.733282 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15689,"thread_start_us":7887,"threads_started":1}
I20260810 23:35:10.733825 31185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:10.708786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.9.48.194:34851)
0810 23:35:10.710164 (+ 1378us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:10.710177 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:10.724670 (+ 14493us) server_negotiation.cc:407] Connection header received
0810 23:35:10.724983 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:10.724992 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:10.725251 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:10.725425 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:10.726455 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.728041 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.730870 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.732309 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.732439 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:10.732721 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:10.733215 (+ 494us) server_negotiation.cc:299] Negotiation successful
0810 23:35:10.733379 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1149,"thread_start_us":937,"threads_started":1}
I20260810 23:35:10.734818 30545 tablet_service.cc:1467] Processing CreateTablet for tablet e7a89480b3ae46229a9d2cd2aea6535f (DEFAULT_TABLE table=test-table [id=21ae41bafd664d3cb383a3af3b65b500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:35:10.735214 31027 tablet_service.cc:1467] Processing CreateTablet for tablet e7a89480b3ae46229a9d2cd2aea6535f (DEFAULT_TABLE table=test-table [id=21ae41bafd664d3cb383a3af3b65b500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:35:10.736353 30545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a89480b3ae46229a9d2cd2aea6535f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:10.736639 31027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a89480b3ae46229a9d2cd2aea6535f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:10.740059 30795 tablet_service.cc:1467] Processing CreateTablet for tablet e7a89480b3ae46229a9d2cd2aea6535f (DEFAULT_TABLE table=test-table [id=21ae41bafd664d3cb383a3af3b65b500]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:35:10.741508 30795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a89480b3ae46229a9d2cd2aea6535f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:10.757638 31188 tablet_bootstrap.cc:492] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0: Bootstrap starting.
I20260810 23:35:10.761596 31187 tablet_bootstrap.cc:492] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3: Bootstrap starting.
I20260810 23:35:10.765401 31188 tablet_bootstrap.cc:654] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:10.766901 31188 log.cc:824] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:10.767663 31187 tablet_bootstrap.cc:654] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:10.768256 31189 tablet_bootstrap.cc:492] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c: Bootstrap starting.
I20260810 23:35:10.769729 31187 log.cc:824] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:10.771757 31189 tablet_bootstrap.cc:654] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:10.773471 31189 log.cc:824] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:10.773542 31187 tablet_bootstrap.cc:492] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3: No bootstrap required, opened a new log
I20260810 23:35:10.773820 31187 ts_tablet_manager.cc:1397] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260810 23:35:10.776408 31188 tablet_bootstrap.cc:492] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0: No bootstrap required, opened a new log
I20260810 23:35:10.776714 31188 ts_tablet_manager.cc:1397] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.003s
I20260810 23:35:10.781311 31187 raft_consensus.cc:348] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46575 } } peers { permanent_uuid: "0c1a7d36c79b45aba51f72eb3d1577c0" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33769 } } peers { permanent_uuid: "3d2c65445d394cc198402508da57aa5c" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34851 } }
I20260810 23:35:10.781890 31187 raft_consensus.cc:374] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:10.782058 31187 raft_consensus.cc:729] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6a4b5cbd924abdb54b04f7fdfe61e3, State: Initialized, Role: FOLLOWER
I20260810 23:35:10.782356 31189 tablet_bootstrap.cc:492] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c: No bootstrap required, opened a new log
I20260810 23:35:10.782627 31189 ts_tablet_manager.cc:1397] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260810 23:35:10.783006 31187 consensus_queue.cc:260] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46575 } } peers { permanent_uuid: "0c1a7d36c79b45aba51f72eb3d1577c0" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33769 } } peers { permanent_uuid: "3d2c65445d394cc198402508da57aa5c" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34851 } }
I20260810 23:35:10.788331 31188 raft_consensus.cc:348] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46575 } } peers { permanent_uuid: "0c1a7d36c79b45aba51f72eb3d1577c0" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33769 } } peers { permanent_uuid: "3d2c65445d394cc198402508da57aa5c" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34851 } }
I20260810 23:35:10.788839 31188 raft_consensus.cc:374] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:10.788977 31188 raft_consensus.cc:729] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1a7d36c79b45aba51f72eb3d1577c0, State: Initialized, Role: FOLLOWER
I20260810 23:35:10.789831 31188 consensus_queue.cc:260] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46575 } } peers { permanent_uuid: "0c1a7d36c79b45aba51f72eb3d1577c0" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33769 } } peers { permanent_uuid: "3d2c65445d394cc198402508da57aa5c" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34851 } }
I20260810 23:35:10.791447 31187 ts_tablet_manager.cc:1428] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
I20260810 23:35:10.791802 31156 heartbeater.cc:503] Master 127.9.48.254:33333 was elected leader, sending a full tablet report...
I20260810 23:35:10.794992 31189 raft_consensus.cc:348] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46575 } } peers { permanent_uuid: "0c1a7d36c79b45aba51f72eb3d1577c0" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33769 } } peers { permanent_uuid: "3d2c65445d394cc198402508da57aa5c" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34851 } }
I20260810 23:35:10.795501 31189 raft_consensus.cc:374] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:10.795647 31189 raft_consensus.cc:729] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d2c65445d394cc198402508da57aa5c, State: Initialized, Role: FOLLOWER
I20260810 23:35:10.796454 31189 consensus_queue.cc:260] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46575 } } peers { permanent_uuid: "0c1a7d36c79b45aba51f72eb3d1577c0" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33769 } } peers { permanent_uuid: "3d2c65445d394cc198402508da57aa5c" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34851 } }
I20260810 23:35:10.799146 30658 heartbeater.cc:503] Master 127.9.48.254:33333 was elected leader, sending a full tablet report...
I20260810 23:35:10.800065 31188 ts_tablet_manager.cc:1428] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0: Time spent starting tablet: real 0.023s user 0.017s sys 0.000s
I20260810 23:35:10.801601 31189 ts_tablet_manager.cc:1428] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c: Time spent starting tablet: real 0.019s user 0.012s sys 0.007s
I20260810 23:35:10.854924 31200 raft_consensus.cc:482] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:35:10.855278 31200 raft_consensus.cc:504] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46575 } } peers { permanent_uuid: "0c1a7d36c79b45aba51f72eb3d1577c0" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33769 } } peers { permanent_uuid: "3d2c65445d394cc198402508da57aa5c" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34851 } }
I20260810 23:35:10.859125 31200 leader_election.cc:288] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b6a4b5cbd924abdb54b04f7fdfe61e3 (127.9.48.195:46575), 0c1a7d36c79b45aba51f72eb3d1577c0 (127.9.48.193:33769)
I20260810 23:35:10.866379 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:10.857884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:50461 (local address 127.9.48.195:46575)
0810 23:35:10.858144 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:10.858152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:10.858842 (+ 690us) server_negotiation.cc:407] Connection header received
0810 23:35:10.859264 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:10.859273 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:10.859389 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:10.859470 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:10.860644 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.861861 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.864037 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.865259 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.865459 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:10.865549 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:10.865883 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 23:35:10.866046 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:35:10.866416 31201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:10.857904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:46575 (local address 127.9.48.194:50461)
0810 23:35:10.858727 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:10.858748 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:10.859013 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:10.859786 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:10.859815 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:10.859905 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:10.860437 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:10.860450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.862037 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.862044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:10.863862 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:10.863875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.865505 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.865511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:10.865682 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:10.865716 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:35:10.865814 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 23:35:10.865952 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"spinlock_wait_cycles":80256,"thread_start_us":417,"threads_started":1}
I20260810 23:35:10.867270 31183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:10.858816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:44629 (local address 127.9.48.193:33769)
0810 23:35:10.859004 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:10.859010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:10.859866 (+ 856us) server_negotiation.cc:407] Connection header received
0810 23:35:10.860020 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:10.860026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:10.860119 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:10.860228 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:10.860929 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.862115 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.864630 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.866030 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.866209 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:10.866287 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:10.866892 (+ 605us) server_negotiation.cc:299] Negotiation successful
0810 23:35:10.867030 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 23:35:10.867228 31157 tablet.cc:2369] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:35:10.868076 31204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:10.858680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:33769 (local address 127.9.48.194:44629)
0810 23:35:10.859719 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:10.859737 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:10.859880 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:10.860260 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:10.860264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:10.860288 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:10.860746 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:10.860758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.862299 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.862311 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:10.864415 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:10.864430 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.866485 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.866495 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:10.866703 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:10.866736 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:35:10.867615 (+ 879us) client_negotiation.cc:240] Negotiation successful
0810 23:35:10.867759 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":749,"threads_started":1}
I20260810 23:35:10.867859 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a89480b3ae46229a9d2cd2aea6535f" candidate_uuid: "3d2c65445d394cc198402508da57aa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3" is_pre_election: true
I20260810 23:35:10.868762 31070 raft_consensus.cc:2454] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d2c65445d394cc198402508da57aa5c in term 0.
I20260810 23:35:10.869387 30578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a89480b3ae46229a9d2cd2aea6535f" candidate_uuid: "3d2c65445d394cc198402508da57aa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1a7d36c79b45aba51f72eb3d1577c0" is_pre_election: true
I20260810 23:35:10.869581 30710 leader_election.cc:302] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d2c65445d394cc198402508da57aa5c, 7b6a4b5cbd924abdb54b04f7fdfe61e3; no voters:
I20260810 23:35:10.869971 30578 raft_consensus.cc:2454] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d2c65445d394cc198402508da57aa5c in term 0.
I20260810 23:35:10.870052 31200 raft_consensus.cc:2781] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:35:10.870270 31200 raft_consensus.cc:482] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:35:10.870412 31200 raft_consensus.cc:3037] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:10.873612 31200 raft_consensus.cc:504] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46575 } } peers { permanent_uuid: "0c1a7d36c79b45aba51f72eb3d1577c0" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33769 } } peers { permanent_uuid: "3d2c65445d394cc198402508da57aa5c" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34851 } }
I20260810 23:35:10.874333 31200 leader_election.cc:288] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [CANDIDATE]: Term 1 election: Requested vote from peers 7b6a4b5cbd924abdb54b04f7fdfe61e3 (127.9.48.195:46575), 0c1a7d36c79b45aba51f72eb3d1577c0 (127.9.48.193:33769)
I20260810 23:35:10.874884 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a89480b3ae46229a9d2cd2aea6535f" candidate_uuid: "3d2c65445d394cc198402508da57aa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3"
I20260810 23:35:10.874977 30578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a89480b3ae46229a9d2cd2aea6535f" candidate_uuid: "3d2c65445d394cc198402508da57aa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1a7d36c79b45aba51f72eb3d1577c0"
I20260810 23:35:10.875162 31070 raft_consensus.cc:3037] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:10.875228 30578 raft_consensus.cc:3037] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:10.879030 31070 raft_consensus.cc:2454] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d2c65445d394cc198402508da57aa5c in term 1.
I20260810 23:35:10.880239 30578 raft_consensus.cc:2454] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d2c65445d394cc198402508da57aa5c in term 1.
I20260810 23:35:10.880889 30711 leader_election.cc:302] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c1a7d36c79b45aba51f72eb3d1577c0, 3d2c65445d394cc198402508da57aa5c; no voters:
I20260810 23:35:10.881385 31200 raft_consensus.cc:2781] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:35:10.882004 31200 raft_consensus.cc:686] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [term 1 LEADER]: Becoming Leader. State: Replica: 3d2c65445d394cc198402508da57aa5c, State: Running, Role: LEADER
I20260810 23:35:10.882756 31200 consensus_queue.cc:237] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46575 } } peers { permanent_uuid: "0c1a7d36c79b45aba51f72eb3d1577c0" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33769 } } peers { permanent_uuid: "3d2c65445d394cc198402508da57aa5c" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34851 } }
I20260810 23:35:10.892040 30320 catalog_manager.cc:5495] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d2c65445d394cc198402508da57aa5c (127.9.48.194). New cstate: current_term: 1 leader_uuid: "3d2c65445d394cc198402508da57aa5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c1a7d36c79b45aba51f72eb3d1577c0" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d2c65445d394cc198402508da57aa5c" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34851 } health_report { overall_health: HEALTHY } } }
W20260810 23:35:10.901818 30916 tablet.cc:2369] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:35:10.920987 30659 tablet.cc:2369] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:35:10.940326 31185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:10.933769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.9.48.194:34851)
0810 23:35:10.933992 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:10.933998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:10.934022 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:35:10.934250 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:10.934256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:10.934388 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:10.934554 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:35:10.934944 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.936049 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.937603 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:10.938240 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:10.938390 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:10.938928 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:35:10.939690 (+ 762us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:35:10.939869 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:10.939941 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 23:35:10.940032 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 23:35:10.950415 31070 raft_consensus.cc:1261] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3 [term 1 FOLLOWER]: Refusing update from remote peer 3d2c65445d394cc198402508da57aa5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:35:10.950485 30578 raft_consensus.cc:1261] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0 [term 1 FOLLOWER]: Refusing update from remote peer 3d2c65445d394cc198402508da57aa5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:35:10.951668 31207 consensus_queue.cc:1035] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1a7d36c79b45aba51f72eb3d1577c0" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:35:10.952188 31200 consensus_queue.cc:1035] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b6a4b5cbd924abdb54b04f7fdfe61e3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:35:10.987293 30319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:35:10.987775 30319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:35:10.991962 30319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e9696f042f343b9b16a23b5dab777de: Sending DeleteTablet for 3 replicas of tablet e7a89480b3ae46229a9d2cd2aea6535f
I20260810 23:35:10.993255 30545 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a89480b3ae46229a9d2cd2aea6535f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:35:10 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:40584
I20260810 23:35:10.993258 31027 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a89480b3ae46229a9d2cd2aea6535f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:35:10 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:53758
I20260810 23:35:10.993386 30795 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a89480b3ae46229a9d2cd2aea6535f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:35:10 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:32982
I20260810 23:35:10.999089 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 30385
I20260810 23:35:11.007848 31232 tablet_replica.cc:330] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0: stopping tablet replica
I20260810 23:35:11.010330 31232 raft_consensus.cc:2229] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:35:11.011021 31232 raft_consensus.cc:2258] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:35:11.013484 31231 tablet_replica.cc:330] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3: stopping tablet replica
I20260810 23:35:11.014038 31231 raft_consensus.cc:2229] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:35:11.015805 31232 ts_tablet_manager.cc:1905] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:35:11.017405 31231 raft_consensus.cc:2258] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:35:11.018088 31233 tablet_replica.cc:330] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c: stopping tablet replica
I20260810 23:35:11.019670 31231 ts_tablet_manager.cc:1905] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:35:11.020370 31233 raft_consensus.cc:2229] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:35:11.021143 31233 raft_consensus.cc:2258] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:35:11.024255 31232 ts_tablet_manager.cc:1918] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:35:11.024492 31232 log.cc:1196] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/e7a89480b3ae46229a9d2cd2aea6535f
I20260810 23:35:11.025245 31232 ts_tablet_manager.cc:1939] T e7a89480b3ae46229a9d2cd2aea6535f P 0c1a7d36c79b45aba51f72eb3d1577c0: Deleting consensus metadata
I20260810 23:35:11.025743 31183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:11.004977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40594 (local address 127.9.48.193:33769)
0810 23:35:11.005222 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:11.005233 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:11.005746 (+ 513us) server_negotiation.cc:407] Connection header received
0810 23:35:11.005973 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:11.005982 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:11.006120 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:11.006241 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:11.008237 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:11.009804 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:11.011677 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:11.012632 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:11.012754 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:11.016857 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:11.016873 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:11.016896 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:11.016942 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:11.020936 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:11.021945 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:11.021954 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:11.021958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:11.022137 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:11.022646 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:11.022653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:11.022656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:11.023157 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:11.023452 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:11.024817 (+ 1365us) server_negotiation.cc:299] Negotiation successful
0810 23:35:11.025096 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:35:11.026772 30648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:11.027554 30294 catalog_manager.cc:4841] TS 0c1a7d36c79b45aba51f72eb3d1577c0 (127.9.48.193:33769): tablet e7a89480b3ae46229a9d2cd2aea6535f (table test-table [id=21ae41bafd664d3cb383a3af3b65b500]) successfully deleted
I20260810 23:35:11.028672 31231 ts_tablet_manager.cc:1918] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:35:11.031584 31233 ts_tablet_manager.cc:1905] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:35:11.032809 31231 log.cc:1196] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/e7a89480b3ae46229a9d2cd2aea6535f
I20260810 23:35:11.033438 31231 ts_tablet_manager.cc:1939] T e7a89480b3ae46229a9d2cd2aea6535f P 7b6a4b5cbd924abdb54b04f7fdfe61e3: Deleting consensus metadata
I20260810 23:35:11.035228 30293 catalog_manager.cc:4841] TS 7b6a4b5cbd924abdb54b04f7fdfe61e3 (127.9.48.195:46575): tablet e7a89480b3ae46229a9d2cd2aea6535f (table test-table [id=21ae41bafd664d3cb383a3af3b65b500]) successfully deleted
I20260810 23:35:11.039296 31233 ts_tablet_manager.cc:1918] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:35:11.039475 31233 log.cc:1196] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/e7a89480b3ae46229a9d2cd2aea6535f
I20260810 23:35:11.040103 31233 ts_tablet_manager.cc:1939] T e7a89480b3ae46229a9d2cd2aea6535f P 3d2c65445d394cc198402508da57aa5c: Deleting consensus metadata
I20260810 23:35:11.042069 30295 catalog_manager.cc:4841] TS 3d2c65445d394cc198402508da57aa5c (127.9.48.194:34851): tablet e7a89480b3ae46229a9d2cd2aea6535f (table test-table [id=21ae41bafd664d3cb383a3af3b65b500]) successfully deleted
I20260810 23:35:11.125556 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 30385
I20260810 23:35:11.162602 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 30664
I20260810 23:35:11.179586 31185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:11.163523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.9.48.194:34851)
0810 23:35:11.163740 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:11.163745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:11.163768 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:35:11.163974 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:11.163978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:11.164066 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:11.164210 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:11.166118 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:11.167049 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:11.169131 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:11.169805 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:11.169936 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:11.173315 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:11.173335 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:11.173357 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:11.173402 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:11.176772 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:11.177609 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:11.177616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:11.177619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:11.177762 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:11.178144 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:11.178147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:11.178149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:11.178499 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:11.178733 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:11.179072 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 23:35:11.179245 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 23:35:11.180408 30902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:11.295908 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 30664
I20260810 23:35:11.322926 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 30920
I20260810 23:35:11.337159 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:11.323719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.9.48.195:46575)
0810 23:35:11.323937 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:11.323943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:11.324048 (+ 105us) server_negotiation.cc:407] Connection header received
0810 23:35:11.324316 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:11.324320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:11.324392 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:11.324516 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:11.325951 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:11.326701 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:11.328027 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:11.328461 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:11.328698 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:11.331276 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:11.331289 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:11.331304 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:11.331338 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:11.334270 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:11.334983 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:11.334999 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:11.335002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:11.335132 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:11.335516 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:11.335521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:11.335526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:11.335877 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:11.336225 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:11.336489 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 23:35:11.336722 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:35:11.337858 31137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:11.429134 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 30920
I20260810 23:35:11.456069 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 30261
I20260810 23:35:11.457350 30374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:11.571368 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 30261
I20260810 23:35:11.607802 9411 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 23:35:11.959228 9411 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 23:35:11.959498 9411 mini_ranger.cc:66] Stopping Ranger...
I20260810 23:35:12.305176 9411 mini_ranger.cc:68] Stopped Ranger
2026-08-10 23:35:12.305 UTC [28701] LOG: received smart shutdown request
2026-08-10 23:35:12.308 UTC [28701] LOG: background worker "logical replication launcher" (PID 28716) exited with exit code 1
2026-08-10 23:35:12.309 UTC [28709] LOG: shutting down
2026-08-10 23:35:12.477 UTC [28701] LOG: database system is shut down
2026-08-10T23:35:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55033 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:35:12 dist-test-slave-04bb krb5kdc[31249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:35:12 dist-test-slave-04bb krb5kdc[31249](info): set up 2 sockets
Aug 10 23:35:12 dist-test-slave-04bb krb5kdc[31249](info): commencing operation
krb5kdc: starting...
W20260810 23:35:13.966275 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.309s user 0.002s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:35:14 dist-test-slave-04bb krb5kdc[31249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:35:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:35:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:35:14.251997 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:40269
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:40269,127.9.48.253:38531,127.9.48.252:39647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:35:14.589347 31274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:14.589624 31274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:14.589826 31274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:14.596894 31274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:35:14.597016 31274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:14.597100 31274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:14.597175 31274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:35:14.597242 31274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:35:14.606752 31274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:40269,127.9.48.253:38531,127.9.48.252:39647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:40269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:14.608071 31274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:14.609395 31274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:14.622084 31286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:14.622162 31284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:14.622174 31283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:14.623000 31274 server_base.cc:1034] running on GCE node
I20260810 23:35:14.623785 31274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:14.625137 31274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:14.626345 31274 hybrid_clock.cc:648] HybridClock initialized: now 1786404914626305 us; error 29 us; skew 500 ppm
I20260810 23:35:14.626703 31274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:14 dist-test-slave-04bb krb5kdc[31249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404914, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:14.634122 31274 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:35:14.635800 31274 webserver.cc:460] Webserver started at http://127.9.48.254:45677/ using document root <none> and password file <none>
I20260810 23:35:14.636643 31274 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:14.636814 31274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:14.637151 31274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:35:14.640400 31274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb"
format_stamp: "Formatted at 2026-08-10 23:35:14 on dist-test-slave-04bb"
server_key: "e64c4d5bedede92acd3fef0e0b2311b7"
server_key_iv: "e6735513a276cbdc5ccb814fbede299f"
server_key_version: "encryptionkey@0"
I20260810 23:35:14.641379 31274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb"
format_stamp: "Formatted at 2026-08-10 23:35:14 on dist-test-slave-04bb"
server_key: "e64c4d5bedede92acd3fef0e0b2311b7"
server_key_iv: "e6735513a276cbdc5ccb814fbede299f"
server_key_version: "encryptionkey@0"
I20260810 23:35:14.646951 31274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:35:14.650821 31297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:14.652163 31274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:35:14.652372 31274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb"
format_stamp: "Formatted at 2026-08-10 23:35:14 on dist-test-slave-04bb"
server_key: "e64c4d5bedede92acd3fef0e0b2311b7"
server_key_iv: "e6735513a276cbdc5ccb814fbede299f"
server_key_version: "encryptionkey@0"
I20260810 23:35:14.652647 31274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:14.676489 31274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:14.681121 31274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:14.681458 31274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:14.701963 31274 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:40269
I20260810 23:35:14.702004 31380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:40269 every 8 connection(s)
I20260810 23:35:14.703470 31274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:35:14.707249 31383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:14.709365 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 31274
I20260810 23:35:14.709637 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:35:14.710513 9411 external_mini_cluster.cc:1428] Setting key cc666771c7c7c300e715c52421093b9d
I20260810 23:35:14.717016 31383 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:35:14 dist-test-slave-04bb krb5kdc[31249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404914, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.48.253@KRBTEST.COM; defaulting to no policy
I20260810 23:35:14.742285 31386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:14.722282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.9.48.254:40269)
0810 23:35:14.722978 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:14.722984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:14.723032 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:35:14.724301 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:14.724306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:14.724578 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:14.724801 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:14.726357 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:14.727571 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:14.729960 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:14.730752 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:14.730891 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:14.734985 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:14.734993 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:14.735006 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:14.735033 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:14.737958 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:14.738652 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:14.738659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:14.738661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:14.738805 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:14.739217 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:14.739227 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:14.739229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:14.739703 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:14.739919 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:14.740464 (+ 545us) server_negotiation.cc:299] Negotiation successful
0810 23:35:14.740801 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":371,"threads_started":1}
I20260810 23:35:14.742329 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:14.720971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40269 (local address 127.0.0.1:58190)
0810 23:35:14.721820 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:14.721877 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:14.723915 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:14.724822 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:14.724828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:14.725624 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:14.726123 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:14.726162 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:14.727749 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:14.727758 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:14.729754 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:14.729772 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:14.731004 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:14.731014 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:14.731285 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:14.732024 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:14.732057 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:14.734754 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:14.738089 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:14.738102 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:14.738114 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:14.738508 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:14.738928 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:14.738933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:14.738937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:14.739099 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:14.739857 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:14.739864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:14.740080 (+ 216us) client_negotiation.cc:769] Sending connection context
0810 23:35:14.740325 (+ 245us) client_negotiation.cc:240] Negotiation successful
0810 23:35:14.740953 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":146,"thread_start_us":371,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.253@KRBTEST.COM" created.
I20260810 23:35:14.745604 31383 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:35:14.746805 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:14.746375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:38531 (local address 127.0.0.1:35090)
0810 23:35:14.746584 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:14.746693 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260810 23:35:14.747123 31304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
I20260810 23:35:14.748783 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:14.748404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:38531 (local address 127.0.0.1:35098)
0810 23:35:14.748602 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:14.748670 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:35:14.748986 31383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.253:38531: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.253.keytab.
Entry for principal kudu/127.9.48.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.253.keytab.
I20260810 23:35:14.776811 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.253:38531
--webserver_interface=127.9.48.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.253.keytab
--principal=kudu/127.9.48.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:40269,127.9.48.253:38531,127.9.48.252:39647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:35:14.798424 31383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } attempt: 1
I20260810 23:35:14.799866 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:14.799228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:38531 (local address 127.0.0.1:35114)
0810 23:35:14.799496 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:14.799634 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 23:35:14.801086 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:14.800638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:38531 (local address 127.0.0.1:35122)
0810 23:35:14.800821 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:14.800917 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:35:14.801301 31383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.253:38531: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
I20260810 23:35:14.869587 31383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } attempt: 2
I20260810 23:35:14.870966 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:14.870358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:38531 (local address 127.0.0.1:35130)
0810 23:35:14.870654 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:14.870774 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 23:35:14.872076 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:14.871668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:38531 (local address 127.0.0.1:35144)
0810 23:35:14.871826 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:14.871923 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 23:35:14.872287 31383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.253:38531: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
I20260810 23:35:14.963584 31383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } attempt: 3
I20260810 23:35:14.965056 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:14.964445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:38531 (local address 127.0.0.1:35150)
0810 23:35:14.964731 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:14.964870 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260810 23:35:14.966149 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:14.965744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:38531 (local address 127.0.0.1:35162)
0810 23:35:14.965909 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:14.965993 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 23:35:14.966344 31383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.253:38531: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
I20260810 23:35:15.109611 31383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } attempt: 4
I20260810 23:35:15.110965 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.110317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:38531 (local address 127.0.0.1:35168)
0810 23:35:15.110665 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.110790 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260810 23:35:15.112059 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.111653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:38531 (local address 127.0.0.1:35170)
0810 23:35:15.111805 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.111907 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:35:15.112285 31383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.253:38531: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
W20260810 23:35:15.114450 31393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:15.114710 31393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:15.114806 31393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:15.121943 31393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:35:15.122038 31393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:15.122172 31393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:15.122264 31393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:35:15.122326 31393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:35:15.132061 31393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:40269,127.9.48.253:38531,127.9.48.252:39647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.253.keytab
--principal=kudu/127.9.48.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.253:38531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.48.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:15.133463 31393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:15.134786 31393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:15.147230 31401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:15.147181 31407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:15.147267 31403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:15.148205 31393 server_base.cc:1034] running on GCE node
I20260810 23:35:15.149121 31393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:15.150584 31393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:15.151818 31393 hybrid_clock.cc:648] HybridClock initialized: now 1786404915151757 us; error 55 us; skew 500 ppm
I20260810 23:35:15.152213 31393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:15 dist-test-slave-04bb krb5kdc[31249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404915, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:15.155601 31393 init.cc:377] Logged in from keytab as kudu/127.9.48.253@KRBTEST.COM (short username kudu)
I20260810 23:35:15.157217 31393 webserver.cc:460] Webserver started at http://127.9.48.253:37311/ using document root <none> and password file <none>
I20260810 23:35:15.157956 31393 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:15.158059 31393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:15.158332 31393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:35:15.161356 31393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/data/instance:
uuid: "25f6fd5fd4c944c4b4779197726d214b"
format_stamp: "Formatted at 2026-08-10 23:35:15 on dist-test-slave-04bb"
server_key: "bb1875df5d3b61020d105afce392b272"
server_key_iv: "fe169eb1a0246ec58715a7683126c17f"
server_key_version: "encryptionkey@0"
I20260810 23:35:15.162091 31393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/wal/instance:
uuid: "25f6fd5fd4c944c4b4779197726d214b"
format_stamp: "Formatted at 2026-08-10 23:35:15 on dist-test-slave-04bb"
server_key: "bb1875df5d3b61020d105afce392b272"
server_key_iv: "fe169eb1a0246ec58715a7683126c17f"
server_key_version: "encryptionkey@0"
I20260810 23:35:15.167158 31393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:35:15.170562 31419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:15.171983 31393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 23:35:15.172158 31393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/wal
uuid: "25f6fd5fd4c944c4b4779197726d214b"
format_stamp: "Formatted at 2026-08-10 23:35:15 on dist-test-slave-04bb"
server_key: "bb1875df5d3b61020d105afce392b272"
server_key_iv: "fe169eb1a0246ec58715a7683126c17f"
server_key_version: "encryptionkey@0"
I20260810 23:35:15.172333 31393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:15.202230 31393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:15.207412 31393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:15.207664 31393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:15.226954 31393 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.253:38531
I20260810 23:35:15.227043 31515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.253:38531 every 8 connection(s)
I20260810 23:35:15.228323 31393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/data/info.pb
I20260810 23:35:15.232249 31516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:15.234180 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 31393
I20260810 23:35:15.234370 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-1/wal/instance
I20260810 23:35:15.234826 9411 external_mini_cluster.cc:1428] Setting key 91325ff577114b28273a70d6c9b89858
I20260810 23:35:15.239316 31516 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:35:15 dist-test-slave-04bb krb5kdc[31249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404915, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.253@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:15.263266 31386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.242516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58200 (local address 127.9.48.254:40269)
0810 23:35:15.242725 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:15.242730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:15.244133 (+ 1403us) server_negotiation.cc:407] Connection header received
0810 23:35:15.246112 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:15.246119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:15.246219 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:15.246347 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:15.247910 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.249108 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.250655 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.251471 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.251644 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.255018 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:15.255028 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:15.255035 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:15.255103 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:15.259462 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.260478 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.260487 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.260492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.260693 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.261160 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.261168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.261174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.261645 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:15.261827 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:15.262511 (+ 684us) server_negotiation.cc:299] Negotiation successful
0810 23:35:15.262775 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:35:15.263829 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.243071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40269 (local address 127.0.0.1:58200)
0810 23:35:15.243908 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.243956 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:15.245817 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:15.246602 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:15.246611 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:15.247131 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:15.247713 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.247731 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.249285 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.249290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.250491 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.250508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.251691 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.251696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.251869 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.252662 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:15.252684 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:15.254781 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:15.259741 (+ 4960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.259747 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.259760 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.260337 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.260825 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.260829 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.260831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.261035 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.261799 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:15.261816 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:15.262130 (+ 314us) client_negotiation.cc:769] Sending connection context
0810 23:35:15.262359 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 23:35:15.262720 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":371,"threads_started":1}
WARNING: no policy specified for kudu/127.9.48.252@KRBTEST.COM; defaulting to no policy
I20260810 23:35:15.265798 31516 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.252@KRBTEST.COM" created.
Aug 10 23:35:15 dist-test-slave-04bb krb5kdc[31249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404915, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.253@KRBTEST.COM for kudu/127.9.48.253@KRBTEST.COM
I20260810 23:35:15.282101 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.266694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:38531 (local address 127.0.0.1:35176)
0810 23:35:15.266881 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.266896 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:15.266979 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:15.267989 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:15.267993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:15.268468 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:15.268679 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.268690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.270076 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.270084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.271540 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.271559 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.272398 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.272407 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.272695 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.273302 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:15.273328 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:15.275922 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:15.279377 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.279385 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.279390 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.279751 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.280140 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.280145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.280148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.280279 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.280939 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:15.280945 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:15.281114 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 23:35:15.281263 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 23:35:15.281598 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 23:35:15.282298 31521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.266848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.9.48.253:38531)
0810 23:35:15.267532 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:15.267538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:15.267563 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:35:15.267603 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:15.267608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:15.267770 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:15.267962 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:15.268867 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.269924 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.271694 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.272232 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.272335 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.276138 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:15.276149 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:15.276162 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:15.276189 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:15.279251 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.279884 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.279892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.279895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.280014 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.280394 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.280400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.280403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.280816 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:15.281032 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:15.281437 (+ 405us) server_negotiation.cc:299] Negotiation successful
0810 23:35:15.281809 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":542,"thread_start_us":395,"threads_started":1}
I20260810 23:35:15.285056 31516 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:35:15.286240 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.285658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:53526)
0810 23:35:15.285872 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.286018 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 23:35:15.286545 31425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
I20260810 23:35:15.288285 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.287861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:53532)
0810 23:35:15.288039 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.288143 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 23:35:15.288573 31516 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:39647: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.252.keytab.
Entry for principal kudu/127.9.48.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.252.keytab.
I20260810 23:35:15.304250 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.252:39647
--webserver_interface=127.9.48.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.252.keytab
--principal=kudu/127.9.48.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:40269,127.9.48.253:38531,127.9.48.252:39647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:35:15.337921 31516 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } attempt: 1
I20260810 23:35:15.339180 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.338690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:53536)
0810 23:35:15.338912 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.339025 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 23:35:15.340354 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.339939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:53540)
0810 23:35:15.340066 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.340147 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 23:35:15.340684 31516 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:39647: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
I20260810 23:35:15.408936 31516 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } attempt: 2
I20260810 23:35:15.410184 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.409731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:32980)
0810 23:35:15.409932 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.410025 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 23:35:15.411178 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.410838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:32988)
0810 23:35:15.410987 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.411064 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 23:35:15.411396 31516 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:39647: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
I20260810 23:35:15.411561 31383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } attempt: 5
Aug 10 23:35:15 dist-test-slave-04bb krb5kdc[31249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404914, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for kudu/127.9.48.253@KRBTEST.COM
I20260810 23:35:15.426276 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.412188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:38531 (local address 127.0.0.1:55448)
0810 23:35:15.412399 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.412420 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:15.412592 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:15.412970 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:15.412974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:15.413374 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:15.413560 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.413570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.415000 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.415004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.415890 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.415909 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.416946 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.416954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.417154 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.417846 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:15.417864 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:15.419861 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:15.423926 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.423930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.423933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.424294 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.424737 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.424740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.424743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.424878 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.425490 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:15.425493 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:15.425607 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 23:35:15.425721 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 23:35:15.425941 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 23:35:15.426406 31521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.412356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55448 (local address 127.9.48.253:38531)
0810 23:35:15.412515 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:15.412556 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:15.412581 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:35:15.412742 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:15.412747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:15.412829 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:15.412939 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:15.413711 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.414856 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.416057 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.416759 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.416890 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.420079 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:15.420101 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:15.420108 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:15.420157 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:15.423783 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.424423 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.424431 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.424435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.424614 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.424984 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.424989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.424992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.425360 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:15.425510 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:15.425863 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 23:35:15.426082 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 23:35:15.427462 31383 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:35:15.428462 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.428088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:33000)
0810 23:35:15.428251 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.428335 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 23:35:15.429913 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.429415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:33006)
0810 23:35:15.429628 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.429749 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 23:35:15.430353 31383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:39647: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
I20260810 23:35:15.481614 31383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } attempt: 1
I20260810 23:35:15.482834 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.482328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:33010)
0810 23:35:15.482535 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.482653 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 23:35:15.483862 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.483532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:33024)
0810 23:35:15.483682 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.483756 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:35:15.484087 31383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:39647: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
I20260810 23:35:15.502703 31516 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } attempt: 3
I20260810 23:35:15.503808 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.503337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:33038)
0810 23:35:15.503573 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.503657 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 23:35:15.504928 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.504465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:33050)
0810 23:35:15.504745 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.504820 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
W20260810 23:35:15.505183 31516 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:39647: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
I20260810 23:35:15.552399 31383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } attempt: 2
I20260810 23:35:15.553607 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.553101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:33056)
0810 23:35:15.553329 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.553443 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 23:35:15.554643 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.554297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:33068)
0810 23:35:15.554462 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.554535 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:35:15.554868 31383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:39647: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
I20260810 23:35:15.648422 31516 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } attempt: 4
W20260810 23:35:15.648948 31525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:15.649271 31525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:15.649437 31525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:35:15.649718 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.649145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:33084)
0810 23:35:15.649411 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.649500 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 23:35:15.650779 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.650431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:33088)
0810 23:35:15.650603 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.650679 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:35:15.650978 31516 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:39647: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
W20260810 23:35:15.656991 31525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:35:15.657114 31525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:15.657213 31525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:15.657313 31525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:35:15.657394 31525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:35:15.661139 31383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } attempt: 3
I20260810 23:35:15.662209 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.661693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:33096)
0810 23:35:15.661912 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.662015 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 23:35:15.663235 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.662853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:33106)
0810 23:35:15.663019 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.663113 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 23:35:15.663440 31383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:39647: Network error: Client connection negotiation failed: client connection to 127.9.48.252:39647: connect: Connection refused (error 111)
I20260810 23:35:15.668188 31525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:40269,127.9.48.253:38531,127.9.48.252:39647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.252.keytab
--principal=kudu/127.9.48.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.252:39647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.48.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:15.669754 31525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:15.671200 31525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:15.683482 31533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:15.683612 31535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:15.683531 31537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:15.684283 31525 server_base.cc:1034] running on GCE node
I20260810 23:35:15.685039 31525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:15.686342 31525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:15.687543 31525 hybrid_clock.cc:648] HybridClock initialized: now 1786404915687479 us; error 53 us; skew 500 ppm
I20260810 23:35:15.687919 31525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:15 dist-test-slave-04bb krb5kdc[31249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404915, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:15.691099 31525 init.cc:377] Logged in from keytab as kudu/127.9.48.252@KRBTEST.COM (short username kudu)
I20260810 23:35:15.692572 31525 webserver.cc:460] Webserver started at http://127.9.48.252:39137/ using document root <none> and password file <none>
I20260810 23:35:15.693279 31525 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:15.693406 31525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:15.693732 31525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:35:15.696770 31525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/data/instance:
uuid: "3a49f65c85154c49bc11353b9c38f336"
format_stamp: "Formatted at 2026-08-10 23:35:15 on dist-test-slave-04bb"
server_key: "65afa0538877addfa539bc11b7edb08d"
server_key_iv: "72304b86dbf51e59082c924df3d8aac2"
server_key_version: "encryptionkey@0"
I20260810 23:35:15.697671 31525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/wal/instance:
uuid: "3a49f65c85154c49bc11353b9c38f336"
format_stamp: "Formatted at 2026-08-10 23:35:15 on dist-test-slave-04bb"
server_key: "65afa0538877addfa539bc11b7edb08d"
server_key_iv: "72304b86dbf51e59082c924df3d8aac2"
server_key_version: "encryptionkey@0"
I20260810 23:35:15.703255 31525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 23:35:15.706784 31544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:15.708263 31525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:35:15.708451 31525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/wal
uuid: "3a49f65c85154c49bc11353b9c38f336"
format_stamp: "Formatted at 2026-08-10 23:35:15 on dist-test-slave-04bb"
server_key: "65afa0538877addfa539bc11b7edb08d"
server_key_iv: "72304b86dbf51e59082c924df3d8aac2"
server_key_version: "encryptionkey@0"
I20260810 23:35:15.708734 31525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:15.723631 31525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:15.727595 31525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:15.727896 31525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:15.746343 31525 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.252:39647
I20260810 23:35:15.746379 31639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.252:39647 every 8 connection(s)
I20260810 23:35:15.747711 31525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/data/info.pb
I20260810 23:35:15.751482 31640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:15.751775 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 31525
I20260810 23:35:15.752005 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786404698557218-9411-0/minicluster-data/master-2/wal/instance
I20260810 23:35:15.752606 9411 external_mini_cluster.cc:1428] Setting key 4f858a79a25d87f58f13963b9dc79aa7
I20260810 23:35:15.761317 31640 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:35:15 dist-test-slave-04bb krb5kdc[31249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.252@KRBTEST.COM
Aug 10 23:35:15 dist-test-slave-04bb krb5kdc[31249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404915, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.252@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:15.779213 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.754647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.9.48.252:39647)
0810 23:35:15.755602 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:15.755613 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:15.755659 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:35:15.758129 (+ 2470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:15.758160 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:15.758551 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:15.758893 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:15.760762 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.762161 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.764070 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.765336 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.765495 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.770572 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:15.770580 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:15.770589 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:15.770647 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:15.774310 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.775134 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.775143 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.775151 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.775361 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.775867 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.775875 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.775879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.776690 (+ 811us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:15.776982 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:15.777533 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 23:35:15.777936 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":500,"threads_started":1}
I20260810 23:35:15.781296 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.764207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40269 (local address 127.0.0.1:37182)
0810 23:35:15.765073 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.765125 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:15.765279 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:15.765713 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:15.765717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:15.766451 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:15.766723 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.766732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.768293 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.768297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.769764 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.769777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.770762 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.770770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.770992 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.771774 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:15.771799 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:15.774659 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:15.778583 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.778593 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.778598 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.779023 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.779501 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.779509 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.779513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.779689 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.780370 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:15.780378 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:15.780588 (+ 210us) client_negotiation.cc:769] Sending connection context
0810 23:35:15.780768 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 23:35:15.780979 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":476,"threads_started":1}
I20260810 23:35:15.781559 31646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.764164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37182 (local address 127.9.48.254:40269)
0810 23:35:15.764726 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:15.764733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:15.765232 (+ 499us) server_negotiation.cc:407] Connection header received
0810 23:35:15.765419 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:15.765425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:15.765512 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:15.765630 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:15.766875 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.768138 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.769932 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.770576 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.770674 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.774851 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:15.774856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:15.774860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:15.774892 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:15.778436 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.779174 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.779182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.779188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.779348 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.779907 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.779913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.779917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.780215 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:15.780336 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:15.780931 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 23:35:15.781172 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":241,"threads_started":1}
I20260810 23:35:15.783069 31640 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:35:15 dist-test-slave-04bb krb5kdc[31249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404915, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.252@KRBTEST.COM for kudu/127.9.48.253@KRBTEST.COM
Aug 10 23:35:15 dist-test-slave-04bb krb5kdc[31249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:15.798251 31521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.783852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.9.48.253:38531)
0810 23:35:15.784049 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:15.784056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:15.784077 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:35:15.784168 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:15.784186 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:15.784289 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:15.784395 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:15.785050 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.786084 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.787830 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.788492 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.788655 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.792604 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:15.792610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:15.792614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:15.792645 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:15.795341 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.796096 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.796100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.796102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.796241 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.796722 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.796729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.796732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.797090 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:15.797253 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:15.797746 (+ 493us) server_negotiation.cc:299] Negotiation successful
0810 23:35:15.797989 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:35:15.798255 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.783819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:38531 (local address 127.0.0.1:55462)
0810 23:35:15.783968 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.783986 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:15.784069 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:15.784419 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:15.784422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:15.784786 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:15.784921 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.784929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.786223 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.786231 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.787640 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.787656 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.789083 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.789088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.789330 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.789775 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:15.789810 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:15.792381 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:15.795479 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.795484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.795488 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.795984 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.796358 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.796364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.796368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.796556 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.797273 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:15.797281 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:15.797471 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 23:35:15.797629 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 23:35:15.797845 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 23:35:15.799425 31640 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:35:15.799579 31646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.782863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.9.48.254:40269)
0810 23:35:15.783046 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:15.783052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:15.783070 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:35:15.783171 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:15.783176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:15.783301 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:15.783415 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:15.784959 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.786005 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.787818 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.788492 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.788658 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.793383 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:15.793397 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:15.793400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:15.793430 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:15.796495 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.797325 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.797330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.797333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.797480 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.797932 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.797938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.797941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.798328 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:15.798576 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:15.799047 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 23:35:15.799283 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 23:35:15 dist-test-slave-04bb krb5kdc[31249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404915, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.252@KRBTEST.COM for kudu/127.9.48.252@KRBTEST.COM
Aug 10 23:35:15 dist-test-slave-04bb krb5kdc[31249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.253@KRBTEST.COM
I20260810 23:35:15.813352 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.800258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.9.48.252:39647)
0810 23:35:15.800417 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:15.800424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:15.800444 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:35:15.800493 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:15.800497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:15.800599 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:15.800703 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:15.801612 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.802649 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.803936 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.804353 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.804444 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.807742 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:15.807748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:15.807753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:15.807788 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:15.810788 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.811401 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.811408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.811411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.811532 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.811884 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.811889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.811892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.812181 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:15.812332 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:15.812722 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 23:35:15.812956 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":34}
I20260810 23:35:15.813524 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.800120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:33120)
0810 23:35:15.800288 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.800306 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:15.800403 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:15.800781 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:15.800788 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:15.801286 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:15.801436 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.801444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.802822 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.802826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.803794 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.803808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.804549 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.804559 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.804717 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.805259 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:15.805275 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:15.807557 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:15.810911 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.810919 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.810922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.811281 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.811637 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.811651 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.811654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.811783 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.812299 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:15.812303 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:15.812472 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 23:35:15.812649 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 23:35:15.813119 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":155}
I20260810 23:35:15.817193 31521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.801593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.9.48.253:38531)
0810 23:35:15.801735 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:15.801746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:15.801909 (+ 163us) server_negotiation.cc:407] Connection header received
0810 23:35:15.802071 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:15.802075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:15.802125 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:15.802211 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:15.803576 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.804397 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.805874 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.806480 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.806575 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.811401 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:15.811406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:15.811409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:15.811438 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:15.814605 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.815240 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.815246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.815250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.815395 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.815781 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.815787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.815790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.816053 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:15.816334 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:15.816671 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 23:35:15.816904 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 23:35:15.818650 31640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336: Bootstrap starting.
I20260810 23:35:15.822333 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:35:15.822638 31640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:15.823694 31640 log.cc:824] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:15.826401 31640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336: No bootstrap required, opened a new log
I20260810 23:35:15.839280 31640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } }
I20260810 23:35:15.839704 31640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:15.839829 31640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a49f65c85154c49bc11353b9c38f336, State: Initialized, Role: FOLLOWER
I20260810 23:35:15.840611 31383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } attempt: 4
I20260810 23:35:15.844810 31640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } }
I20260810 23:35:15.848681 31670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } } }
I20260810 23:35:15.849241 31670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:35:15.850641 31640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:35:15.852393 31646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.828047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.9.48.254:40269)
0810 23:35:15.828288 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:15.828297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:15.829049 (+ 752us) server_negotiation.cc:407] Connection header received
0810 23:35:15.829400 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:15.829409 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:15.829519 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:15.829608 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:15.832701 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.834181 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.837138 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.838033 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.838154 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.842800 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:15.842819 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:15.842824 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:15.842869 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:15.846637 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.849677 (+ 3040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.849684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.849688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.849865 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.850425 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.850432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.850436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.850797 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:15.851083 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:15.851729 (+ 646us) server_negotiation.cc:299] Negotiation successful
0810 23:35:15.852000 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:35:15.857033 31521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.827598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55486 (local address 127.9.48.253:38531)
0810 23:35:15.827787 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:15.827793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:15.835940 (+ 8147us) server_negotiation.cc:407] Connection header received
0810 23:35:15.836358 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:15.836364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:15.836439 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:15.836506 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:15.839028 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.840115 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.842497 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.843355 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.843472 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.850144 (+ 6672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:15.850152 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:15.850160 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:15.850209 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:15.854088 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.854929 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.854935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.854938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.855074 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.855473 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.855478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.855481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.855774 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:15.856042 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:15.856417 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 23:35:15.856764 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:35:15.861284 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.828977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33130 (local address 127.9.48.252:39647)
0810 23:35:15.829165 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:15.829171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:15.840240 (+ 11069us) server_negotiation.cc:407] Connection header received
0810 23:35:15.840589 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:15.840596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:15.840675 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:15.840746 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:15.842949 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.844088 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.846396 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.847127 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.847235 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.853752 (+ 6517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:15.853763 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:15.853770 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:15.853821 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:15.858501 (+ 4680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.859135 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.859141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.859144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.859297 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.859694 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.859699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.859702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.859991 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:15.860273 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:15.860658 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 23:35:15.860899 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 23:35:15 dist-test-slave-04bb krb5kdc[31249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404914, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for kudu/127.9.48.252@KRBTEST.COM
I20260810 23:35:15.876210 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.841140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:33146)
0810 23:35:15.841454 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.841473 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:15.841572 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:15.848477 (+ 6905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:15.848484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:15.848983 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:15.849179 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.849191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.860725 (+ 11534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.860734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.862531 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.862556 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.863906 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.863910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.864044 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.864657 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:15.864675 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:15.866740 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:15.871467 (+ 4727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.871473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.871477 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.872045 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.872603 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.872607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.872612 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.872799 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.874749 (+ 1950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:15.874755 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:15.874917 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 23:35:15.875563 (+ 646us) client_negotiation.cc:240] Negotiation successful
0810 23:35:15.875915 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204}
I20260810 23:35:15.876770 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.841764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33146 (local address 127.9.48.252:39647)
0810 23:35:15.848157 (+ 6393us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:15.848163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:15.848187 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:35:15.848241 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:15.848248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:15.848328 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:15.848447 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:15.858765 (+ 10318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.860467 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.862791 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.863748 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.863901 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.867031 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:15.867040 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:15.867047 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:15.867085 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:15.871313 (+ 4228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.872193 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.872198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.872201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.872395 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.872947 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.872952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.872955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.873380 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:15.873523 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:15.875230 (+ 1707us) server_negotiation.cc:299] Negotiation successful
0810 23:35:15.875453 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6241,"thread_start_us":6051,"threads_started":1}
W20260810 23:35:15.875243 31689 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:35:15.883194 31689 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:35:15.885215 31383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb: Bootstrap starting.
I20260810 23:35:15.888401 31383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:15.871773 31525 master_runner.cc:186] Error getting master registration for 127.9.48.254:40269: OK, instance_id { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" instance_seqno: 1786404914696110 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:35:15.888749 31525 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:35:15.889863 31383 log.cc:824] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:15.892688 31383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb: No bootstrap required, opened a new log
I20260810 23:35:15.900751 31383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } }
I20260810 23:35:15.901041 31383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:15.901203 31383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a90f6562a2b42d3a19bf2a3e8faa3bb, State: Initialized, Role: FOLLOWER
I20260810 23:35:15.901858 31383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } }
I20260810 23:35:15.902871 31694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } } }
I20260810 23:35:15.903271 31694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:35:15.904119 31383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:35:15.916729 31709 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:35:15.916860 31709 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:35:15.916100 31274 master_runner.cc:186] Error getting master registration for 127.9.48.253:38531: OK, instance_id { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" instance_seqno: 1786404915221559 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:35:15.928421 31274 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:35:15.934700 31309 tablet.cc:2369] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:35:15.950205 31516 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } attempt: 5
Aug 10 23:35:15 dist-test-slave-04bb krb5kdc[31249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404915, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.253@KRBTEST.COM for kudu/127.9.48.252@KRBTEST.COM
I20260810 23:35:15.970453 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.950940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.9.48.252:39647)
0810 23:35:15.951120 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:15.951127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:15.951192 (+ 65us) server_negotiation.cc:407] Connection header received
0810 23:35:15.951398 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:15.951403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:15.951508 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:15.951612 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:15.952713 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.953897 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.955669 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.956378 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.956509 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.963129 (+ 6620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:15.963137 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:15.963151 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:15.963213 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:15.967241 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.968136 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.968150 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.968153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.968333 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:15.968791 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:15.968796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:15.968800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:15.969201 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:15.969356 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:15.969832 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 23:35:15.970072 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:35:15.970448 31518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:15.950854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:39647 (local address 127.0.0.1:33162)
0810 23:35:15.951116 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:15.951136 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:15.951258 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:15.951685 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:15.951691 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:15.952259 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:15.952482 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.952495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.954053 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.954059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.955504 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:15.955523 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:15.956566 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:15.956573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:15.956863 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:15.957585 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:15.957613 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:15.962890 (+ 5277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:15.967408 (+ 4518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.967413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.967418 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.967997 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.968461 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:15.968466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:15.968470 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:15.968667 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:15.969339 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:15.969345 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:15.969516 (+ 171us) client_negotiation.cc:769] Sending connection context
0810 23:35:15.969673 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 23:35:15.969990 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260810 23:35:15.977048 31516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b: Bootstrap starting.
W20260810 23:35:15.980198 31559 tablet.cc:2369] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:35:15.982645 31516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:15.984135 31516 log.cc:824] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:15.987648 31516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b: No bootstrap required, opened a new log
I20260810 23:35:15.999104 31516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } }
I20260810 23:35:15.999603 31516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:15.999756 31516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f6fd5fd4c944c4b4779197726d214b, State: Initialized, Role: FOLLOWER
I20260810 23:35:16.000510 31516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } }
I20260810 23:35:16.001571 31713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } } }
I20260810 23:35:16.001955 31713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:35:16.003050 31516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:35:16.019557 31393 master_runner.cc:263] No leader master found from master 25f6fd5fd4c944c4b4779197726d214b
I20260810 23:35:16.019723 31393 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:35:16.026360 31730 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:35:16.026616 31730 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:35:16.059993 31670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:35:16.060316 31670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } }
I20260810 23:35:16.062013 31670 leader_election.cc:288] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a90f6562a2b42d3a19bf2a3e8faa3bb (127.9.48.254:40269), 25f6fd5fd4c944c4b4779197726d214b (127.9.48.253:38531)
I20260810 23:35:16.063022 31465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a49f65c85154c49bc11353b9c38f336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25f6fd5fd4c944c4b4779197726d214b" is_pre_election: true
I20260810 23:35:16.063658 31465 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a49f65c85154c49bc11353b9c38f336 in term 0.
I20260810 23:35:16.064316 31555 leader_election.cc:302] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25f6fd5fd4c944c4b4779197726d214b, 3a49f65c85154c49bc11353b9c38f336; no voters:
I20260810 23:35:16.065523 31338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a49f65c85154c49bc11353b9c38f336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" is_pre_election: true
I20260810 23:35:16.066140 31338 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a49f65c85154c49bc11353b9c38f336 in term 0.
I20260810 23:35:16.067000 31670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:35:16.067225 31670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:35:16.067346 31670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:16.070561 31670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } }
I20260810 23:35:16.072698 31338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a49f65c85154c49bc11353b9c38f336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb"
I20260810 23:35:16.072942 31338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:16.076537 31338 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a49f65c85154c49bc11353b9c38f336 in term 1.
I20260810 23:35:16.076964 31553 leader_election.cc:302] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a49f65c85154c49bc11353b9c38f336, 5a90f6562a2b42d3a19bf2a3e8faa3bb; no voters:
I20260810 23:35:16.079491 31465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a49f65c85154c49bc11353b9c38f336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25f6fd5fd4c944c4b4779197726d214b"
I20260810 23:35:16.079733 31465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:16.081579 31670 leader_election.cc:288] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [CANDIDATE]: Term 1 election: Requested vote from peers 5a90f6562a2b42d3a19bf2a3e8faa3bb (127.9.48.254:40269), 25f6fd5fd4c944c4b4779197726d214b (127.9.48.253:38531)
I20260810 23:35:16.081820 31670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:35:16.081970 31670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [term 1 LEADER]: Becoming Leader. State: Replica: 3a49f65c85154c49bc11353b9c38f336, State: Running, Role: LEADER
I20260810 23:35:16.082584 31670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } }
I20260810 23:35:16.083330 31465 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a49f65c85154c49bc11353b9c38f336 in term 1.
I20260810 23:35:16.115904 31732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a49f65c85154c49bc11353b9c38f336. Latest consensus state: current_term: 1 leader_uuid: "3a49f65c85154c49bc11353b9c38f336" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } } }
I20260810 23:35:16.116405 31732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:16.129261 31735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:35:16.135948 31735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:35:16.159253 31338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb [term 1 FOLLOWER]: Refusing update from remote peer 3a49f65c85154c49bc11353b9c38f336: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:35:16.159300 31465 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b [term 1 FOLLOWER]: Refusing update from remote peer 3a49f65c85154c49bc11353b9c38f336: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:35:16.160331 31670 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:35:16.161468 31732 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:35:16.209602 31694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a49f65c85154c49bc11353b9c38f336. Latest consensus state: current_term: 1 leader_uuid: "3a49f65c85154c49bc11353b9c38f336" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } } }
I20260810 23:35:16.209956 31694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:35:16.210351 31427 tablet.cc:2369] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:35:16.216058 31740 mvcc.cc:205] Tried to move back new op lower bound from 7317114536571187200 to 7317114536289095680. Current Snapshot: MvccSnapshot[applied={T|T < 7317114536571187200}]
I20260810 23:35:16.219267 31713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a49f65c85154c49bc11353b9c38f336. Latest consensus state: current_term: 1 leader_uuid: "3a49f65c85154c49bc11353b9c38f336" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } } }
I20260810 23:35:16.219609 31713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:35:16.221489 31670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a49f65c85154c49bc11353b9c38f336" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } } }
I20260810 23:35:16.221828 31670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:16.223443 31735 catalog_manager.cc:1347] Generated new cluster ID: 2bf919ffd3324f48926671b3aa1f2b30
I20260810 23:35:16.223565 31735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:35:16.236935 31670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a49f65c85154c49bc11353b9c38f336" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } } }
I20260810 23:35:16.237242 31670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:16.237769 31694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a49f65c85154c49bc11353b9c38f336" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } } }
I20260810 23:35:16.238029 31694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:35:16.239883 31743 mvcc.cc:205] Tried to move back new op lower bound from 7317114536571187200 to 7317114536289095680. Current Snapshot: MvccSnapshot[applied={T|T < 7317114536571187200}]
I20260810 23:35:16.256922 31713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a49f65c85154c49bc11353b9c38f336" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a90f6562a2b42d3a19bf2a3e8faa3bb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40269 } } peers { permanent_uuid: "25f6fd5fd4c944c4b4779197726d214b" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 38531 } } peers { permanent_uuid: "3a49f65c85154c49bc11353b9c38f336" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 39647 } } }
I20260810 23:35:16.257203 31713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:35:16.266131 31735 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:35:16.268105 31735 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:35:16.289940 31735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336: Generated new TSK 0
I20260810 23:35:16.291103 31735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:35:16.918942 31709 catalog_manager.cc:1259] Loaded cluster ID: 2bf919ffd3324f48926671b3aa1f2b30
I20260810 23:35:16.919116 31709 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb: loading cluster ID for follower catalog manager: success
I20260810 23:35:16.922272 31709 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb: acquiring CA information for follower catalog manager: success
I20260810 23:35:16.924067 31709 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5a90f6562a2b42d3a19bf2a3e8faa3bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:35:17.028744 31730 catalog_manager.cc:1259] Loaded cluster ID: 2bf919ffd3324f48926671b3aa1f2b30
I20260810 23:35:17.028856 31730 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b: loading cluster ID for follower catalog manager: success
I20260810 23:35:17.031294 31730 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b: acquiring CA information for follower catalog manager: success
I20260810 23:35:17.033102 31730 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25f6fd5fd4c944c4b4779197726d214b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:35:17.440419 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 31274
I20260810 23:35:17.441636 31372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:17.553225 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 31274
I20260810 23:35:17.585129 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 31393
I20260810 23:35:17.586334 31505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:17.698033 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 31393
I20260810 23:35:17.707275 31767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:17.705432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40269 (local address 127.0.0.1:37196)
0810 23:35:17.706972 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:17.707098 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1413,"thread_start_us":1282,"threads_started":1}
W20260810 23:35:17.707566 31553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40269: connect: Connection refused (error 111)
I20260810 23:35:17.709256 31767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:17.708893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40269 (local address 127.0.0.1:37202)
0810 23:35:17.709047 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:17.709125 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 23:35:17.709478 31553 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 -> Peer 5a90f6562a2b42d3a19bf2a3e8faa3bb (127.9.48.254:40269): Couldn't send request to peer 5a90f6562a2b42d3a19bf2a3e8faa3bb. Status: Network error: Client connection negotiation failed: client connection to 127.9.48.254:40269: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260810 23:35:17.730319 31555 connection.cc:535] client connection to 127.9.48.253:38531 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:35:17.730710 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 31525
I20260810 23:35:17.731668 31767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:17.731153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:38531 (local address 127.0.0.1:55502)
0810 23:35:17.731394 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:17.731467 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260810 23:35:17.731925 31632 generic_service.cc:196] Checking for leaks (request via RPC)
W20260810 23:35:17.731925 31555 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3a49f65c85154c49bc11353b9c38f336 -> Peer 25f6fd5fd4c944c4b4779197726d214b (127.9.48.253:38531): Couldn't send request to peer 25f6fd5fd4c944c4b4779197726d214b. Status: Network error: Client connection negotiation failed: client connection to 127.9.48.253:38531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:35:17.842933 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 31525
2026-08-10T23:35:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5236 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:35:17 dist-test-slave-04bb krb5kdc[31775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:35:17 dist-test-slave-04bb krb5kdc[31775](info): set up 2 sockets
Aug 10 23:35:17 dist-test-slave-04bb krb5kdc[31775](info): commencing operation
krb5kdc: starting...
W20260810 23:35:19.246518 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.353s user 0.005s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:35:19 dist-test-slave-04bb krb5kdc[31775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:35:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:35:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:35:19.544147 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:44751
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:39391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:44751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:35:19.887394 31801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:19.887776 31801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:19.887967 31801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:19.895581 31801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:35:19.895725 31801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:19.895843 31801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:19.895946 31801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:35:19.896034 31801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:35:19.906155 31801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:39391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:44751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:44751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:19.907582 31801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:19.909080 31801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:19.921470 31809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:19.921541 31813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:19.921517 31810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:19.922458 31801 server_base.cc:1034] running on GCE node
I20260810 23:35:19.923239 31801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:19.924695 31801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:19.925947 31801 hybrid_clock.cc:648] HybridClock initialized: now 1786404919925896 us; error 57 us; skew 500 ppm
I20260810 23:35:19.926349 31801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:19 dist-test-slave-04bb krb5kdc[31775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404919, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:19.933712 31801 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:35:19.935253 31801 webserver.cc:460] Webserver started at http://127.9.48.254:37793/ using document root <none> and password file <none>
I20260810 23:35:19.935920 31801 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:19.936017 31801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:19.936287 31801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:35:19.939294 31801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "68003e15f29e4206addfee7c8a144e57"
format_stamp: "Formatted at 2026-08-10 23:35:19 on dist-test-slave-04bb"
server_key: "cd6aa4ae9d2e1f7def38aee10f78c453"
server_key_iv: "ce231e54f0ae5ab904c411028c542608"
server_key_version: "encryptionkey@0"
I20260810 23:35:19.939985 31801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "68003e15f29e4206addfee7c8a144e57"
format_stamp: "Formatted at 2026-08-10 23:35:19 on dist-test-slave-04bb"
server_key: "cd6aa4ae9d2e1f7def38aee10f78c453"
server_key_iv: "ce231e54f0ae5ab904c411028c542608"
server_key_version: "encryptionkey@0"
I20260810 23:35:19.945083 31801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:35:19.948464 31825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:19.949821 31801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:35:19.949955 31801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "68003e15f29e4206addfee7c8a144e57"
format_stamp: "Formatted at 2026-08-10 23:35:19 on dist-test-slave-04bb"
server_key: "cd6aa4ae9d2e1f7def38aee10f78c453"
server_key_iv: "ce231e54f0ae5ab904c411028c542608"
server_key_version: "encryptionkey@0"
I20260810 23:35:19.950119 31801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:19.966825 31801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:19.970964 31801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:19.971220 31801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:19.990968 31801 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:44751
I20260810 23:35:19.991035 31910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:44751 every 8 connection(s)
I20260810 23:35:19.992390 31801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:35:19.993896 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 31801
I20260810 23:35:19.994223 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:35:19.995242 9411 external_mini_cluster.cc:1428] Setting key e7408e84b7043557c51284cb2552ee79
I20260810 23:35:19.997684 31911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:35:20 dist-test-slave-04bb krb5kdc[31775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:20.012835 31911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57: Bootstrap starting.
I20260810 23:35:20.017583 31911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:20.019069 31911 log.cc:824] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:20.022218 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:19.997273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49320 (local address 127.9.48.254:44751)
0810 23:35:19.998087 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:19.998099 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:19.998142 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:35:19.999791 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:19.999812 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:20.000178 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:20.000513 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:20.002691 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:20.004051 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:20.006152 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:20.006819 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:20.006945 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:20.012418 (+ 5473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:20.012431 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:20.012452 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:20.012515 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:20.016999 (+ 4484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:20.017817 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:20.017829 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:20.017838 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:20.018121 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:20.018575 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:20.018581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:20.018585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:20.019428 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:20.019821 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:20.020258 (+ 437us) server_negotiation.cc:299] Negotiation successful
0810 23:35:20.020719 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":386,"threads_started":1}
I20260810 23:35:20.022969 31911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57: No bootstrap required, opened a new log
I20260810 23:35:20.026367 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:35:20.032025 31911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68003e15f29e4206addfee7c8a144e57" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44751 } }
I20260810 23:35:20.032428 31911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:20.032591 31911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68003e15f29e4206addfee7c8a144e57, State: Initialized, Role: FOLLOWER
I20260810 23:35:20.033262 31911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68003e15f29e4206addfee7c8a144e57" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44751 } }
I20260810 23:35:20.033527 31911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:35:20.033686 31911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:35:20.033900 31911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:20.036249 31911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68003e15f29e4206addfee7c8a144e57" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44751 } }
I20260810 23:35:20.036811 31911 leader_election.cc:302] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68003e15f29e4206addfee7c8a144e57; no voters:
I20260810 23:35:20.037477 31911 leader_election.cc:288] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:35:20.037627 31917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:35:20.038447 31917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [term 1 LEADER]: Becoming Leader. State: Replica: 68003e15f29e4206addfee7c8a144e57, State: Running, Role: LEADER
I20260810 23:35:20.039064 31917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68003e15f29e4206addfee7c8a144e57" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44751 } }
I20260810 23:35:20.039976 31911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:35:20.041910 31921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68003e15f29e4206addfee7c8a144e57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68003e15f29e4206addfee7c8a144e57" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44751 } } }
I20260810 23:35:20.041929 31922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68003e15f29e4206addfee7c8a144e57. Latest consensus state: current_term: 1 leader_uuid: "68003e15f29e4206addfee7c8a144e57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68003e15f29e4206addfee7c8a144e57" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44751 } } }
I20260810 23:35:20.042384 31922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:20.042384 31921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57 [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:20.044160 31932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:35:20.049788 31932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:35:20.064908 31932 catalog_manager.cc:1347] Generated new cluster ID: 3e848f2083cd4762814421a38736511d
I20260810 23:35:20.065056 31932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:35:20.090765 31932 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:35:20.092831 31932 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:35:20.106585 31932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68003e15f29e4206addfee7c8a144e57: Generated new TSK 0
I20260810 23:35:20.107668 31932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:35:20.139417 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 31801
I20260810 23:35:20.140743 31904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:20.237536 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 31801
2026-08-10T23:35:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2390 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:35:20 dist-test-slave-04bb krb5kdc[31962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:35:20 dist-test-slave-04bb krb5kdc[31962](info): set up 2 sockets
Aug 10 23:35:20 dist-test-slave-04bb krb5kdc[31962](info): commencing operation
krb5kdc: starting...
W20260810 23:35:21.598049 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.313s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:35:21 dist-test-slave-04bb krb5kdc[31962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:35:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:35:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:35:21.900774 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:33443
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:33443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:35:22.248781 31994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:22.249135 31994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:22.249307 31994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:22.256693 31994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:35:22.256858 31994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:22.256965 31994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:22.257071 31994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:35:22.257156 31994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:35:22.267174 31994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:33443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:33443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:22.268667 31994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:22.270018 31994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:22.281905 32002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:22.281848 32003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:22.281807 32008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:22.283041 31994 server_base.cc:1034] running on GCE node
I20260810 23:35:22.283905 31994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:22.285449 31994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:22.286684 31994 hybrid_clock.cc:648] HybridClock initialized: now 1786404922286632 us; error 30 us; skew 500 ppm
I20260810 23:35:22.287022 31994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:22 dist-test-slave-04bb krb5kdc[31962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404922, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:22.290418 31994 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:35:22.291905 31994 webserver.cc:460] Webserver started at http://127.9.48.254:36043/ using document root <none> and password file <none>
I20260810 23:35:22.292655 31994 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:22.292765 31994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:22.293036 31994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:35:22.296092 31994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5"
format_stamp: "Formatted at 2026-08-10 23:35:22 on dist-test-slave-04bb"
server_key: "6f19b43ca1a0893c59b79551201a36a4"
server_key_iv: "275bc01dcf2e410a9e4e2021125a1c98"
server_key_version: "encryptionkey@0"
I20260810 23:35:22.296900 31994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5"
format_stamp: "Formatted at 2026-08-10 23:35:22 on dist-test-slave-04bb"
server_key: "6f19b43ca1a0893c59b79551201a36a4"
server_key_iv: "275bc01dcf2e410a9e4e2021125a1c98"
server_key_version: "encryptionkey@0"
I20260810 23:35:22.302178 31994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 23:35:22.305691 32026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:22.307044 31994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:35:22.307184 31994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5"
format_stamp: "Formatted at 2026-08-10 23:35:22 on dist-test-slave-04bb"
server_key: "6f19b43ca1a0893c59b79551201a36a4"
server_key_iv: "275bc01dcf2e410a9e4e2021125a1c98"
server_key_version: "encryptionkey@0"
I20260810 23:35:22.307346 31994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:22.346768 31994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:22.351370 31994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:22.351629 31994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:22.371809 31994 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:33443
I20260810 23:35:22.371855 32113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:33443 every 8 connection(s)
I20260810 23:35:22.373342 31994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:35:22.377307 32114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:22.380445 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 31994
I20260810 23:35:22.380705 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:35:22.381412 9411 external_mini_cluster.cc:1428] Setting key 45339e168b8aa316739dbf7b0a301c8e
I20260810 23:35:22.390908 32114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: Bootstrap starting.
I20260810 23:35:22.394604 32114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:22.395634 32114 log.cc:824] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:35:22 dist-test-slave-04bb krb5kdc[31962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:22.399061 32114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: No bootstrap required, opened a new log
I20260810 23:35:22.407920 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:22.383608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.9.48.254:33443)
0810 23:35:22.384442 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:22.384459 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:22.384511 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:35:22.386489 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:22.386519 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:22.387056 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:22.387495 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:22.388931 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:22.390945 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:22.392376 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:22.393446 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:22.393657 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:22.398903 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:22.398916 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:22.398935 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:22.398983 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:22.403227 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:22.403935 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:22.403948 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:22.403956 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:22.404178 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:22.404578 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:22.404585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:22.404590 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:22.405355 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:22.405678 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:22.406039 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 23:35:22.406510 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":353,"threads_started":1}
I20260810 23:35:22.410364 32114 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } }
I20260810 23:35:22.410903 32114 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:22.411029 32114 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2dd8b84b8a4baeb717cb50a8b1a9f5, State: Initialized, Role: FOLLOWER
I20260810 23:35:22.411687 32114 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } }
I20260810 23:35:22.411938 32114 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:35:22.412089 32114 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:35:22.412288 32114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:22.414580 32114 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } }
I20260810 23:35:22.415102 32114 leader_election.cc:302] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd2dd8b84b8a4baeb717cb50a8b1a9f5; no voters:
I20260810 23:35:22.415721 32114 leader_election.cc:288] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:35:22.415868 32120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:35:22.417541 32120 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 1 LEADER]: Becoming Leader. State: Replica: cd2dd8b84b8a4baeb717cb50a8b1a9f5, State: Running, Role: LEADER
I20260810 23:35:22.418129 32120 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } }
I20260810 23:35:22.418673 32114 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:35:22.421221 32127 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd2dd8b84b8a4baeb717cb50a8b1a9f5. Latest consensus state: current_term: 1 leader_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } } }
I20260810 23:35:22.421603 32127 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:22.422132 32122 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } } }
I20260810 23:35:22.422403 32122 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:22.424782 32141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:35:22.430483 32141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:35:22.439623 32141 catalog_manager.cc:1347] Generated new cluster ID: f0bf2c2a8bce4c03a9ab121ec9e243f9
I20260810 23:35:22.439746 32141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:35:22.451426 32141 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:35:22.452872 32141 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:35:22.463174 32141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: Generated new TSK 0
I20260810 23:35:22.464279 32141 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:35:22.536774 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33443
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:35:22.885680 32162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:22.885994 32162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:22.886206 32162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:22.893772 32162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:22.893896 32162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:22.894119 32162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:35:22.904453 32162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:22.905833 32162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:22.907473 32162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:22.922696 32168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:22.922765 32169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:22.922675 32171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:22.923468 32162 server_base.cc:1034] running on GCE node
I20260810 23:35:22.924261 32162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:22.925467 32162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:22.926708 32162 hybrid_clock.cc:648] HybridClock initialized: now 1786404922926613 us; error 80 us; skew 500 ppm
I20260810 23:35:22.927053 32162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:22 dist-test-slave-04bb krb5kdc[31962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404922, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:22.930416 32162 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:35:22.932040 32162 webserver.cc:460] Webserver started at http://127.9.48.193:37739/ using document root <none> and password file <none>
I20260810 23:35:22.932837 32162 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:22.932988 32162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:22.933305 32162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:35:22.936331 32162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "bc9342795a77415eb176245baee56fac"
format_stamp: "Formatted at 2026-08-10 23:35:22 on dist-test-slave-04bb"
server_key: "c3e6c766be2cabe52fddf20c459c86c8"
server_key_iv: "3d65d795585b65f8f6df0fec91ceac53"
server_key_version: "encryptionkey@0"
I20260810 23:35:22.937176 32162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "bc9342795a77415eb176245baee56fac"
format_stamp: "Formatted at 2026-08-10 23:35:22 on dist-test-slave-04bb"
server_key: "c3e6c766be2cabe52fddf20c459c86c8"
server_key_iv: "3d65d795585b65f8f6df0fec91ceac53"
server_key_version: "encryptionkey@0"
I20260810 23:35:22.942539 32162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:35:22.946226 32183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:22.947589 32162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:35:22.947773 32162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "bc9342795a77415eb176245baee56fac"
format_stamp: "Formatted at 2026-08-10 23:35:22 on dist-test-slave-04bb"
server_key: "c3e6c766be2cabe52fddf20c459c86c8"
server_key_iv: "3d65d795585b65f8f6df0fec91ceac53"
server_key_version: "encryptionkey@0"
I20260810 23:35:22.947994 32162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:22.971122 32162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:22.975720 32162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:22.976008 32162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:22.977042 32162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:22.978391 32162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:22.978504 32162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:22.978652 32162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:22.978724 32162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:23.016126 32162 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:46731
I20260810 23:35:23.016178 32354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:46731 every 8 connection(s)
I20260810 23:35:23.017558 32162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:35:23.026825 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 32162
I20260810 23:35:23.027063 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:35:23.027783 9411 external_mini_cluster.cc:1428] Setting key e9cced4c940681cf05f7d8266fb6ace2
Aug 10 23:35:23 dist-test-slave-04bb krb5kdc[31962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404922, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:23.039201 32357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:23.020305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:53845 (local address 127.9.48.254:33443)
0810 23:35:23.020808 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:23.020813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:23.021795 (+ 982us) server_negotiation.cc:407] Connection header received
0810 23:35:23.023662 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:23.023668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:23.023784 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:23.023907 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:23.025501 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:23.026231 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:23.027923 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:23.028332 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:23.028489 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:23.032582 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:23.032589 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:23.032593 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:23.032643 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:23.035747 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:23.036606 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:23.036611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:23.036613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:23.036749 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:23.037252 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:23.037257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:23.037259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:23.037548 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:23.037738 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:23.038507 (+ 769us) server_negotiation.cc:299] Negotiation successful
0810 23:35:23.038773 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":256,"threads_started":1}
I20260810 23:35:23.040268 32358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:23.020901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:53845)
0810 23:35:23.021625 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:23.021665 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:23.023382 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:23.024158 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:23.024166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:23.024754 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:23.025327 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:23.025345 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:23.026427 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:23.026431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:23.027757 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:23.027770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:23.028627 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:23.028649 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:23.028963 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:23.029716 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:23.029756 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:23.032339 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:23.035915 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:23.035926 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:23.035943 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:23.036415 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:23.036882 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:23.036888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:23.036892 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:23.037103 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:23.037706 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:23.037714 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:23.038082 (+ 368us) client_negotiation.cc:769] Sending connection context
0810 23:35:23.038386 (+ 304us) client_negotiation.cc:240] Negotiation successful
0810 23:35:23.038975 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":287,"threads_started":1}
I20260810 23:35:23.042027 32355 heartbeater.cc:347] Connected to a master server at 127.9.48.254:33443
I20260810 23:35:23.042423 32355 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:23.043375 32355 heartbeater.cc:511] Master 127.9.48.254:33443 requested a full tablet report, sending...
I20260810 23:35:23.045560 32054 ts_manager.cc:194] Registered new tserver with Master: bc9342795a77415eb176245baee56fac (127.9.48.193:46731)
I20260810 23:35:23.047039 32054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:53845
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:35:23.098183 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33443
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:35:23.460275 32362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:23.460688 32362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:23.460862 32362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:23.468712 32362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:23.468855 32362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:23.469043 32362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:35:23.478967 32362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:23.480341 32362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:23.481758 32362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:23.495743 32369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:23.495692 32376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:23.495767 32372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:23.496558 32362 server_base.cc:1034] running on GCE node
I20260810 23:35:23.497195 32362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:23.498106 32362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:23.499326 32362 hybrid_clock.cc:648] HybridClock initialized: now 1786404923499276 us; error 41 us; skew 500 ppm
I20260810 23:35:23.499600 32362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:23 dist-test-slave-04bb krb5kdc[31962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404923, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:23.502851 32362 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:35:23.504464 32362 webserver.cc:460] Webserver started at http://127.9.48.194:40889/ using document root <none> and password file <none>
I20260810 23:35:23.505278 32362 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:23.505443 32362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:23.505800 32362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:35:23.509205 32362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "2a26c78b6a3647c7a9ae32aaea540fc1"
format_stamp: "Formatted at 2026-08-10 23:35:23 on dist-test-slave-04bb"
server_key: "95cc3b97e57ad531d52ea22180fc3420"
server_key_iv: "e9ef9422a60c0af396a45a3471fdd0d3"
server_key_version: "encryptionkey@0"
I20260810 23:35:23.510092 32362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "2a26c78b6a3647c7a9ae32aaea540fc1"
format_stamp: "Formatted at 2026-08-10 23:35:23 on dist-test-slave-04bb"
server_key: "95cc3b97e57ad531d52ea22180fc3420"
server_key_iv: "e9ef9422a60c0af396a45a3471fdd0d3"
server_key_version: "encryptionkey@0"
I20260810 23:35:23.516040 32362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 23:35:23.520457 32386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:23.522048 32362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:35:23.522266 32362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "2a26c78b6a3647c7a9ae32aaea540fc1"
format_stamp: "Formatted at 2026-08-10 23:35:23 on dist-test-slave-04bb"
server_key: "95cc3b97e57ad531d52ea22180fc3420"
server_key_iv: "e9ef9422a60c0af396a45a3471fdd0d3"
server_key_version: "encryptionkey@0"
I20260810 23:35:23.522514 32362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:23.553903 32362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:23.558380 32362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:23.558717 32362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:23.559767 32362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:23.561326 32362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:23.561440 32362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:23.561610 32362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:23.561694 32362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:23.598155 32362 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:35525
I20260810 23:35:23.598198 32560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:35525 every 8 connection(s)
I20260810 23:35:23.599684 32362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:35:23.608945 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 32362
I20260810 23:35:23.609148 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:35:23.609634 9411 external_mini_cluster.cc:1428] Setting key bfe611bdcf50ff1bff04880baad61e0a
Aug 10 23:35:23 dist-test-slave-04bb krb5kdc[31962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404923, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:23.623536 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:23.602440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:32881 (local address 127.9.48.254:33443)
0810 23:35:23.602917 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:23.602922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:23.603942 (+ 1020us) server_negotiation.cc:407] Connection header received
0810 23:35:23.606082 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:23.606088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:23.606198 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:23.606333 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:23.607982 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:23.608854 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:23.610976 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:23.611565 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:23.611729 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:23.616148 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:23.616157 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:23.616164 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:23.616215 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:23.619968 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:23.620872 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:23.620880 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:23.620886 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:23.621063 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:23.621505 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:23.621512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:23.621517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:23.621927 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:23.622082 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:23.622794 (+ 712us) server_negotiation.cc:299] Negotiation successful
0810 23:35:23.623057 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":245,"threads_started":1}
I20260810 23:35:23.624352 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:23.603015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:32881)
0810 23:35:23.603762 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:23.603805 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:23.605783 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:23.606600 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:23.606610 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:23.607142 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:23.607780 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:23.607799 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:23.609036 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:23.609040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:23.610791 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:23.610805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:23.611825 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:23.611835 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:23.612163 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:23.612967 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:23.613003 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:23.615913 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:23.620122 (+ 4209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:23.620129 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:23.620142 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:23.620744 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:23.621177 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:23.621181 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:23.621183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:23.621370 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:23.622067 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:23.622073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:23.622394 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 23:35:23.622641 (+ 247us) client_negotiation.cc:240] Negotiation successful
0810 23:35:23.623079 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":293,"threads_started":1}
I20260810 23:35:23.625978 32562 heartbeater.cc:347] Connected to a master server at 127.9.48.254:33443
I20260810 23:35:23.626363 32562 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:23.627382 32562 heartbeater.cc:511] Master 127.9.48.254:33443 requested a full tablet report, sending...
I20260810 23:35:23.629041 32054 ts_manager.cc:194] Registered new tserver with Master: 2a26c78b6a3647c7a9ae32aaea540fc1 (127.9.48.194:35525)
I20260810 23:35:23.629869 32054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:32881
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:35:23.678023 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33443
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:35:24.030340 32579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:24.030645 32579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:24.030871 32579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:24.038357 32579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:24.038502 32579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:24.038708 32579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:35:24.048993 32579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:24.050420 32579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:24.051066 32355 heartbeater.cc:503] Master 127.9.48.254:33443 was elected leader, sending a full tablet report...
I20260810 23:35:24.052232 32579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:24.066064 32586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:24.066123 32589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:24.066236 32585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:24.067060 32579 server_base.cc:1034] running on GCE node
I20260810 23:35:24.067673 32579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:24.068723 32579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:24.069949 32579 hybrid_clock.cc:648] HybridClock initialized: now 1786404924069896 us; error 40 us; skew 500 ppm
I20260810 23:35:24.070300 32579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:24 dist-test-slave-04bb krb5kdc[31962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404924, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:24.073748 32579 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:35:24.075317 32579 webserver.cc:460] Webserver started at http://127.9.48.195:38001/ using document root <none> and password file <none>
I20260810 23:35:24.076036 32579 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:24.076159 32579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:24.076437 32579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:35:24.079550 32579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "f7b37e3c16c74680add62a67795f6781"
format_stamp: "Formatted at 2026-08-10 23:35:24 on dist-test-slave-04bb"
server_key: "73eaa0960890c6456ebb4ed5f3a03656"
server_key_iv: "ce76dc43567412c91743d3ef02d0a9ab"
server_key_version: "encryptionkey@0"
I20260810 23:35:24.080492 32579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "f7b37e3c16c74680add62a67795f6781"
format_stamp: "Formatted at 2026-08-10 23:35:24 on dist-test-slave-04bb"
server_key: "73eaa0960890c6456ebb4ed5f3a03656"
server_key_iv: "ce76dc43567412c91743d3ef02d0a9ab"
server_key_version: "encryptionkey@0"
I20260810 23:35:24.086055 32579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 23:35:24.089694 32599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:24.091267 32579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:35:24.091418 32579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "f7b37e3c16c74680add62a67795f6781"
format_stamp: "Formatted at 2026-08-10 23:35:24 on dist-test-slave-04bb"
server_key: "73eaa0960890c6456ebb4ed5f3a03656"
server_key_iv: "ce76dc43567412c91743d3ef02d0a9ab"
server_key_version: "encryptionkey@0"
I20260810 23:35:24.091661 32579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:24.114703 32579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:24.118966 32579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:24.119271 32579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:24.120337 32579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:24.121767 32579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:24.121876 32579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:24.122035 32579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:24.122110 32579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:24.156899 32579 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:39501
I20260810 23:35:24.157015 32766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:39501 every 8 connection(s)
I20260810 23:35:24.158241 32579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:35:24.167450 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 32579
I20260810 23:35:24.167652 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:35:24.168139 9411 external_mini_cluster.cc:1428] Setting key 59c08abc22baec6f449164ffd98a1c7c
Aug 10 23:35:24 dist-test-slave-04bb krb5kdc[31962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404924, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:24.179225 303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.160979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:35037 (local address 127.9.48.254:33443)
0810 23:35:24.161474 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:24.161482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:24.162470 (+ 988us) server_negotiation.cc:407] Connection header received
0810 23:35:24.164338 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:24.164344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:24.164443 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:24.164594 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:24.166085 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.166923 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.168688 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.169083 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.169247 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.172986 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:24.172991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:24.172995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:24.173028 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:24.175780 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:24.176629 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:24.176635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:24.176641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:24.176792 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:24.177298 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:24.177303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:24.177305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:24.177598 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:24.177789 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:24.178512 (+ 723us) server_negotiation.cc:299] Negotiation successful
0810 23:35:24.178775 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":236,"threads_started":1}
I20260810 23:35:24.180207 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.161553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:35037)
0810 23:35:24.162304 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:24.162342 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:24.164051 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:24.164836 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:24.164844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:24.165343 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:24.165906 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:24.165922 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.167097 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.167102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:24.168474 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:24.168486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.169336 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.169341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:24.169655 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.170372 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:24.170399 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:24.172772 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:24.175949 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:24.175959 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:24.175976 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:24.176449 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:24.176924 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:24.176929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:24.176934 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:24.177144 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:24.177764 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:24.177773 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:24.178123 (+ 350us) client_negotiation.cc:769] Sending connection context
0810 23:35:24.178414 (+ 291us) client_negotiation.cc:240] Negotiation successful
0810 23:35:24.179013 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":304,"threads_started":1}
I20260810 23:35:24.181695 32767 heartbeater.cc:347] Connected to a master server at 127.9.48.254:33443
I20260810 23:35:24.182046 32767 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:24.182931 32767 heartbeater.cc:511] Master 127.9.48.254:33443 requested a full tablet report, sending...
I20260810 23:35:24.184593 32054 ts_manager.cc:194] Registered new tserver with Master: f7b37e3c16c74680add62a67795f6781 (127.9.48.195:39501)
I20260810 23:35:24.185497 32054 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:35037
I20260810 23:35:24.196313 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:35:24.214120 303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.201263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.9.48.254:33443)
0810 23:35:24.201435 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:24.201440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:24.201936 (+ 496us) server_negotiation.cc:407] Connection header received
0810 23:35:24.202074 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:24.202077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:24.202140 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:24.202226 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:24.203573 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.204312 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.205638 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.206039 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.206137 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.208595 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:24.208601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:24.208605 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:24.208642 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:24.211388 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:24.212130 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:24.212135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:24.212137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:24.212271 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:24.212690 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:24.212694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:24.212696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:24.213059 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:24.213265 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:24.213651 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 23:35:24.213837 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:35:24.218271 32054 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:35:24.226080 32054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:35:24.254537 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.238825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.9.48.194:35525)
0810 23:35:24.241250 (+ 2425us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:24.241258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:24.241286 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:35:24.241384 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:24.241392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:24.242976 (+ 1584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:24.244192 (+ 1216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:24.244513 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.247971 (+ 3458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.251998 (+ 4027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.253366 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.253529 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.253755 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:24.254019 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 23:35:24.254216 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2269,"thread_start_us":2160,"threads_started":1}
I20260810 23:35:24.254550 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.238555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:35525 (local address 127.0.0.1:49460)
0810 23:35:24.240325 (+ 1770us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:24.240392 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:24.240636 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:24.243282 (+ 2646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:24.243293 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:24.243348 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:24.243917 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:24.243934 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.248147 (+ 4213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.248155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:24.251813 (+ 3658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:24.251830 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.253578 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.253584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:24.253765 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.253813 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 23:35:24.253948 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 23:35:24.254160 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1436,"thread_start_us":12933,"threads_started":1}
I20260810 23:35:24.257526 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.239050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46731 (local address 127.0.0.1:60290)
0810 23:35:24.248126 (+ 9076us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:24.248151 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:24.248278 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:24.249458 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:24.249465 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:24.249493 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:24.249947 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:24.249961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.251934 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.251941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:24.254044 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:24.254058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.256716 (+ 2658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.256723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:24.256917 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.256952 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:35:24.257049 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:35:24.257170 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8920,"thread_start_us":855,"threads_started":1}
I20260810 23:35:24.257602 329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.239742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60290 (local address 127.9.48.193:46731)
0810 23:35:24.248519 (+ 8777us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:24.248547 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:24.248577 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:35:24.248679 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:24.248686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:24.248946 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:24.249152 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:24.250157 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.251791 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.254213 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.255707 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.255883 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.256122 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:24.257124 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0810 23:35:24.257294 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8582,"thread_start_us":8390,"threads_started":1}
I20260810 23:35:24.258927 32461 tablet_service.cc:1467] Processing CreateTablet for tablet e7cf2e546eaa426e898cec7edc7f2983 (DEFAULT_TABLE table=test-table [id=6d3b9ed3dbe04775b3139e15ac01ebc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:35:24.260349 325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.241864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:39501 (local address 127.0.0.1:42064)
0810 23:35:24.251816 (+ 9952us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:24.251836 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:24.251954 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:24.252731 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:24.252736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:24.252762 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:24.253199 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:24.253210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.255893 (+ 2683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.255898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:24.257815 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:24.257830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.259552 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.259558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:24.259725 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.259784 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 23:35:24.259880 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 23:35:24.260011 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9826,"thread_start_us":5919,"threads_started":1}
I20260810 23:35:24.260682 32461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7cf2e546eaa426e898cec7edc7f2983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:24.261121 331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.242288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.9.48.195:39501)
0810 23:35:24.251360 (+ 9072us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:24.251372 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:24.252011 (+ 639us) server_negotiation.cc:407] Connection header received
0810 23:35:24.252124 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:24.252131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:24.252375 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:24.252611 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:24.253412 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.255769 (+ 2357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.257990 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.259336 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.259471 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.259720 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:24.260630 (+ 910us) server_negotiation.cc:299] Negotiation successful
0810 23:35:24.260792 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8882,"thread_start_us":8698,"threads_started":1}
I20260810 23:35:24.261168 32252 tablet_service.cc:1467] Processing CreateTablet for tablet e7cf2e546eaa426e898cec7edc7f2983 (DEFAULT_TABLE table=test-table [id=6d3b9ed3dbe04775b3139e15ac01ebc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:35:24.262422 32252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7cf2e546eaa426e898cec7edc7f2983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:24.268261 32667 tablet_service.cc:1467] Processing CreateTablet for tablet e7cf2e546eaa426e898cec7edc7f2983 (DEFAULT_TABLE table=test-table [id=6d3b9ed3dbe04775b3139e15ac01ebc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:35:24.269826 32667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7cf2e546eaa426e898cec7edc7f2983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:24.273576 332 tablet_bootstrap.cc:492] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac: Bootstrap starting.
I20260810 23:35:24.275588 333 tablet_bootstrap.cc:492] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1: Bootstrap starting.
I20260810 23:35:24.277537 332 tablet_bootstrap.cc:654] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:24.278864 332 log.cc:824] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:24.279241 333 tablet_bootstrap.cc:654] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:24.280354 333 log.cc:824] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:24.282065 332 tablet_bootstrap.cc:492] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac: No bootstrap required, opened a new log
I20260810 23:35:24.282361 332 ts_tablet_manager.cc:1397] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260810 23:35:24.283529 333 tablet_bootstrap.cc:492] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1: No bootstrap required, opened a new log
I20260810 23:35:24.283892 333 ts_tablet_manager.cc:1397] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 23:35:24.283959 337 tablet_bootstrap.cc:492] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781: Bootstrap starting.
I20260810 23:35:24.286788 337 tablet_bootstrap.cc:654] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:24.288342 337 log.cc:824] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:24.290941 337 tablet_bootstrap.cc:492] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781: No bootstrap required, opened a new log
I20260810 23:35:24.291229 337 ts_tablet_manager.cc:1397] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260810 23:35:24.293565 332 raft_consensus.cc:348] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } }
I20260810 23:35:24.294008 332 raft_consensus.cc:374] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:24.294154 332 raft_consensus.cc:729] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc9342795a77415eb176245baee56fac, State: Initialized, Role: FOLLOWER
I20260810 23:35:24.294701 333 raft_consensus.cc:348] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } }
I20260810 23:35:24.294896 332 consensus_queue.cc:260] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } }
I20260810 23:35:24.295151 333 raft_consensus.cc:374] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:24.295289 333 raft_consensus.cc:729] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a26c78b6a3647c7a9ae32aaea540fc1, State: Initialized, Role: FOLLOWER
I20260810 23:35:24.296105 333 consensus_queue.cc:260] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } }
I20260810 23:35:24.298233 332 ts_tablet_manager.cc:1428] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac: Time spent starting tablet: real 0.016s user 0.008s sys 0.009s
I20260810 23:35:24.299218 32562 heartbeater.cc:503] Master 127.9.48.254:33443 was elected leader, sending a full tablet report...
I20260810 23:35:24.300165 333 ts_tablet_manager.cc:1428] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260810 23:35:24.300271 337 raft_consensus.cc:348] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } }
I20260810 23:35:24.300825 337 raft_consensus.cc:374] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:24.301625 337 raft_consensus.cc:729] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b37e3c16c74680add62a67795f6781, State: Initialized, Role: FOLLOWER
I20260810 23:35:24.302253 337 consensus_queue.cc:260] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } }
I20260810 23:35:24.303419 32767 heartbeater.cc:503] Master 127.9.48.254:33443 was elected leader, sending a full tablet report...
I20260810 23:35:24.304023 337 ts_tablet_manager.cc:1428] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
W20260810 23:35:24.351910 32564 tablet.cc:2369] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:35:24.390556 339 raft_consensus.cc:482] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:35:24.390926 339 raft_consensus.cc:504] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } }
I20260810 23:35:24.392961 339 leader_election.cc:288] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a26c78b6a3647c7a9ae32aaea540fc1 (127.9.48.194:35525), f7b37e3c16c74680add62a67795f6781 (127.9.48.195:39501)
I20260810 23:35:24.401778 331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.393993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:51925 (local address 127.9.48.195:39501)
0810 23:35:24.394228 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:24.394237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:24.395047 (+ 810us) server_negotiation.cc:407] Connection header received
0810 23:35:24.395363 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:24.395368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:24.395431 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:24.395533 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:24.396177 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.397470 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.399698 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.400815 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.401008 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.401089 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:24.401395 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 23:35:24.401521 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:35:24.401854 343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.393523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:39501 (local address 127.9.48.193:51925)
0810 23:35:24.394929 (+ 1406us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:24.394968 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:24.395212 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:24.395554 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:24.395558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:24.395592 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:24.396022 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:24.396036 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.397624 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.397629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:24.399539 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:24.399553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.401047 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.401054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:24.401221 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.401247 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:35:24.401355 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 23:35:24.401485 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1252,"thread_start_us":946,"threads_started":1}
I20260810 23:35:24.403294 32700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7cf2e546eaa426e898cec7edc7f2983" candidate_uuid: "bc9342795a77415eb176245baee56fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7b37e3c16c74680add62a67795f6781" is_pre_election: true
I20260810 23:35:24.403934 32700 raft_consensus.cc:2454] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc9342795a77415eb176245baee56fac in term 0.
I20260810 23:35:24.404846 32185 leader_election.cc:302] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc9342795a77415eb176245baee56fac, f7b37e3c16c74680add62a67795f6781; no voters:
I20260810 23:35:24.405437 339 raft_consensus.cc:2781] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:35:24.405743 339 raft_consensus.cc:482] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:35:24.405913 339 raft_consensus.cc:3037] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:24.409195 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.393744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:40589 (local address 127.9.48.194:35525)
0810 23:35:24.393969 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:24.393976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:24.402187 (+ 8211us) server_negotiation.cc:407] Connection header received
0810 23:35:24.402496 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:24.402502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:24.402594 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:24.402663 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:24.403362 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.404388 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.406677 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.407965 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.408088 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.408169 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:24.408802 (+ 633us) server_negotiation.cc:299] Negotiation successful
0810 23:35:24.408941 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:35:24.409871 343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.393727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:35525 (local address 127.9.48.193:40589)
0810 23:35:24.402085 (+ 8358us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:24.402103 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:24.402308 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:24.402953 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:24.402958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:24.402984 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:24.403180 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:24.403190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.404615 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.404619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:24.406466 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:24.406487 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.408312 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.408318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:24.408594 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.408629 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:35:24.409389 (+ 760us) client_negotiation.cc:240] Negotiation successful
0810 23:35:24.409536 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8267,"thread_start_us":14681,"threads_started":1}
W20260810 23:35:24.409921 301 tablet.cc:2369] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:35:24.411324 339 raft_consensus.cc:504] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } }
I20260810 23:35:24.411626 32489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7cf2e546eaa426e898cec7edc7f2983" candidate_uuid: "bc9342795a77415eb176245baee56fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" is_pre_election: true
I20260810 23:35:24.412262 32489 raft_consensus.cc:2454] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc9342795a77415eb176245baee56fac in term 0.
I20260810 23:35:24.412595 32488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7cf2e546eaa426e898cec7edc7f2983" candidate_uuid: "bc9342795a77415eb176245baee56fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1"
I20260810 23:35:24.412902 32488 raft_consensus.cc:3037] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:24.412985 32700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7cf2e546eaa426e898cec7edc7f2983" candidate_uuid: "bc9342795a77415eb176245baee56fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7b37e3c16c74680add62a67795f6781"
I20260810 23:35:24.413228 32700 raft_consensus.cc:3037] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:24.415386 32700 raft_consensus.cc:2454] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc9342795a77415eb176245baee56fac in term 1.
I20260810 23:35:24.415632 339 leader_election.cc:288] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [CANDIDATE]: Term 1 election: Requested vote from peers 2a26c78b6a3647c7a9ae32aaea540fc1 (127.9.48.194:35525), f7b37e3c16c74680add62a67795f6781 (127.9.48.195:39501)
I20260810 23:35:24.415870 32185 leader_election.cc:302] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc9342795a77415eb176245baee56fac, f7b37e3c16c74680add62a67795f6781; no voters:
I20260810 23:35:24.416740 339 raft_consensus.cc:2781] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:35:24.416829 32488 raft_consensus.cc:2454] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc9342795a77415eb176245baee56fac in term 1.
I20260810 23:35:24.417234 339 raft_consensus.cc:686] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [term 1 LEADER]: Becoming Leader. State: Replica: bc9342795a77415eb176245baee56fac, State: Running, Role: LEADER
I20260810 23:35:24.417696 339 consensus_queue.cc:237] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } }
I20260810 23:35:24.422505 32052 catalog_manager.cc:5495] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac reported cstate change: term changed from 0 to 1, leader changed from <none> to bc9342795a77415eb176245baee56fac (127.9.48.193). New cstate: current_term: 1 leader_uuid: "bc9342795a77415eb176245baee56fac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } health_report { overall_health: UNKNOWN } } }
I20260810 23:35:24.443912 329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.437078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.9.48.193:46731)
0810 23:35:24.437294 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:24.437299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:24.437318 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:35:24.437580 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:24.437584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:24.437689 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:24.437820 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:35:24.438350 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.439063 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.441418 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.441926 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.442055 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.442717 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:35:24.443283 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:35:24.443404 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:24.443563 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 23:35:24.443639 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:35:24.454622 32700 raft_consensus.cc:1261] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781 [term 1 FOLLOWER]: Refusing update from remote peer bc9342795a77415eb176245baee56fac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:35:24.454622 32488 raft_consensus.cc:1261] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 1 FOLLOWER]: Refusing update from remote peer bc9342795a77415eb176245baee56fac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:35:24.455762 346 consensus_queue.cc:1035] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:35:24.456120 339 consensus_queue.cc:1035] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:35:24.473735 356 mvcc.cc:205] Tried to move back new op lower bound from 7317114570554404864 to 7317114570418962432. Current Snapshot: MvccSnapshot[applied={T|T < 7317114570554404864}]
I20260810 23:35:24.499984 32052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:35:24.500275 32052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:35:24.503808 32052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: Sending DeleteTablet for 3 replicas of tablet e7cf2e546eaa426e898cec7edc7f2983
I20260810 23:35:24.504971 32461 tablet_service.cc:1514] Processing DeleteTablet for tablet e7cf2e546eaa426e898cec7edc7f2983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:35:24 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:49460
I20260810 23:35:24.505220 32667 tablet_service.cc:1514] Processing DeleteTablet for tablet e7cf2e546eaa426e898cec7edc7f2983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:35:24 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:42064
I20260810 23:35:24.505775 32252 tablet_service.cc:1514] Processing DeleteTablet for tablet e7cf2e546eaa426e898cec7edc7f2983 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:35:24 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:60290
I20260810 23:35:24.506899 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 32162
I20260810 23:35:24.513082 371 tablet_replica.cc:330] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1: stopping tablet replica
I20260810 23:35:24.513365 373 tablet_replica.cc:330] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac: stopping tablet replica
I20260810 23:35:24.513795 371 raft_consensus.cc:2229] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:35:24.513908 373 raft_consensus.cc:2229] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:35:24.514868 373 raft_consensus.cc:2258] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:35:24.515797 371 raft_consensus.cc:2258] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:35:24.517164 373 ts_tablet_manager.cc:1905] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:35:24.520040 371 ts_tablet_manager.cc:1905] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:35:24.522174 372 tablet_replica.cc:330] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781: stopping tablet replica
I20260810 23:35:24.522650 372 raft_consensus.cc:2229] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:35:24.523157 372 raft_consensus.cc:2258] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:35:24.525677 373 ts_tablet_manager.cc:1918] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:35:24.525883 373 log.cc:1196] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/e7cf2e546eaa426e898cec7edc7f2983
I20260810 23:35:24.526504 373 ts_tablet_manager.cc:1939] T e7cf2e546eaa426e898cec7edc7f2983 P bc9342795a77415eb176245baee56fac: Deleting consensus metadata
I20260810 23:35:24.528200 372 ts_tablet_manager.cc:1905] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:35:24.528398 32031 catalog_manager.cc:4841] TS bc9342795a77415eb176245baee56fac (127.9.48.193:46731): tablet e7cf2e546eaa426e898cec7edc7f2983 (table test-table [id=6d3b9ed3dbe04775b3139e15ac01ebc3]) successfully deleted
Aug 10 23:35:24 dist-test-slave-04bb krb5kdc[31962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:35:24.535223 372 ts_tablet_manager.cc:1918] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:35:24.535401 372 log.cc:1196] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/e7cf2e546eaa426e898cec7edc7f2983
I20260810 23:35:24.535966 372 ts_tablet_manager.cc:1939] T e7cf2e546eaa426e898cec7edc7f2983 P f7b37e3c16c74680add62a67795f6781: Deleting consensus metadata
I20260810 23:35:24.537855 32031 catalog_manager.cc:4841] TS f7b37e3c16c74680add62a67795f6781 (127.9.48.195:39501): tablet e7cf2e546eaa426e898cec7edc7f2983 (table test-table [id=6d3b9ed3dbe04775b3139e15ac01ebc3]) successfully deleted
I20260810 23:35:24.540865 329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.508366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.9.48.193:46731)
0810 23:35:24.508600 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:24.508610 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:24.521404 (+ 12794us) server_negotiation.cc:407] Connection header received
0810 23:35:24.521898 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:24.521904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:24.521993 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:24.522070 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:24.524665 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.525825 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.528908 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.529415 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.529559 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.534147 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:24.534159 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:24.534173 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:24.534210 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:24.537521 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:24.538532 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:24.538540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:24.538544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:24.538719 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:24.539180 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:24.539185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:24.539188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:24.539536 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:24.539861 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:24.540195 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 23:35:24.540448 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:35:24.544035 371 ts_tablet_manager.cc:1918] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:35:24.544199 371 log.cc:1196] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/e7cf2e546eaa426e898cec7edc7f2983
I20260810 23:35:24.544816 371 ts_tablet_manager.cc:1939] T e7cf2e546eaa426e898cec7edc7f2983 P 2a26c78b6a3647c7a9ae32aaea540fc1: Deleting consensus metadata
I20260810 23:35:24.546557 32032 catalog_manager.cc:4841] TS 2a26c78b6a3647c7a9ae32aaea540fc1 (127.9.48.194:35525): tablet e7cf2e546eaa426e898cec7edc7f2983 (table test-table [id=6d3b9ed3dbe04775b3139e15ac01ebc3]) successfully deleted
I20260810 23:35:24.549194 32350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:24.662139 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 32162
I20260810 23:35:24.689951 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 32362
Aug 10 23:35:24 dist-test-slave-04bb krb5kdc[31962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:35:24.705782 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.690821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.9.48.194:35525)
0810 23:35:24.691009 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:24.691014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:24.691033 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:35:24.691161 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:24.691165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:24.691246 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:24.691346 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:24.692786 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.693532 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.694810 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.695244 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.695349 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.700326 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:24.700342 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:24.700358 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:24.700397 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:24.703158 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:24.703799 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:24.703806 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:24.703809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:24.703939 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:24.704257 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:24.704261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:24.704263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:24.704847 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:24.705085 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:24.705350 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 23:35:24.705512 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:35:24.706543 32555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:24.796998 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 32362
I20260810 23:35:24.827186 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 32579
Aug 10 23:35:24 dist-test-slave-04bb krb5kdc[31962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:35:24.843557 331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:24.828160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.9.48.195:39501)
0810 23:35:24.828377 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:24.828383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:24.828404 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:35:24.828633 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:24.828640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:24.828714 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:24.828810 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:24.830197 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.830979 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.832202 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:24.832747 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:24.832855 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:24.837380 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:24.837396 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:24.837414 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:24.837455 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:24.840272 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:24.841190 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:24.841199 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:24.841202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:24.841368 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:24.841797 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:24.841803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:24.841807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:24.842211 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:24.842564 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:24.842928 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 23:35:24.843191 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:35:24.844312 32761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:24.934165 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 32579
I20260810 23:35:24.957553 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 31994
I20260810 23:35:24.958813 32108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:25.071062 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 31994
I20260810 23:35:25.105195 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:33443
--webserver_interface=127.9.48.254
--webserver_port=36043
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:33443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:35:25.449671 379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:25.450013 379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:25.450168 379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:25.457314 379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:35:25.457444 379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:25.457552 379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:25.457656 379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:35:25.457742 379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:35:25.467379 379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:33443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:33443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=36043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:25.468747 379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:25.470106 379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:25.501142 390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:25.501132 395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:25.501292 391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:25.501960 379 server_base.cc:1034] running on GCE node
I20260810 23:35:25.502938 379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:25.504770 379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:25.506016 379 hybrid_clock.cc:648] HybridClock initialized: now 1786404925505957 us; error 64 us; skew 500 ppm
I20260810 23:35:25.506438 379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:25 dist-test-slave-04bb krb5kdc[31962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404925, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:25.515503 379 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:35:25.517344 379 webserver.cc:460] Webserver started at http://127.9.48.254:36043/ using document root <none> and password file <none>
I20260810 23:35:25.518167 379 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:25.518289 379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:25.524655 379 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:35:25.527741 403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:25.529342 379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:35:25.529541 379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5"
format_stamp: "Formatted at 2026-08-10 23:35:22 on dist-test-slave-04bb"
server_key: "6f19b43ca1a0893c59b79551201a36a4"
server_key_iv: "275bc01dcf2e410a9e4e2021125a1c98"
server_key_version: "encryptionkey@0"
I20260810 23:35:25.530652 379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:25.549222 379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:25.553907 379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:25.554219 379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:25.574414 379 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:33443
I20260810 23:35:25.574440 488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:33443 every 8 connection(s)
I20260810 23:35:25.575969 379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:35:25.583562 489 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:35:25.585843 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 379
I20260810 23:35:25.586681 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:46731
--local_ip_for_outbound_sockets=127.9.48.193
--tserver_master_addrs=127.9.48.254:33443
--webserver_port=37739
--webserver_interface=127.9.48.193
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:35:25.587365 489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: Bootstrap starting.
I20260810 23:35:25.609797 489 log.cc:824] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:25.623487 489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:35:25.624100 489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: Bootstrap complete.
I20260810 23:35:25.634719 489 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } }
I20260810 23:35:25.635779 489 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2dd8b84b8a4baeb717cb50a8b1a9f5, State: Initialized, Role: FOLLOWER
I20260810 23:35:25.636457 489 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } }
I20260810 23:35:25.636794 489 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:35:25.636961 489 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:35:25.637171 489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:35:25.641500 489 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } }
I20260810 23:35:25.642098 489 leader_election.cc:302] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd2dd8b84b8a4baeb717cb50a8b1a9f5; no voters:
I20260810 23:35:25.642822 489 leader_election.cc:288] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:35:25.642985 493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:35:25.644418 493 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 2 LEADER]: Becoming Leader. State: Replica: cd2dd8b84b8a4baeb717cb50a8b1a9f5, State: Running, Role: LEADER
I20260810 23:35:25.645203 493 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } }
I20260810 23:35:25.645648 489 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:35:25.648202 496 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd2dd8b84b8a4baeb717cb50a8b1a9f5. Latest consensus state: current_term: 2 leader_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } } }
I20260810 23:35:25.648644 496 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:25.648306 495 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } } }
I20260810 23:35:25.649065 495 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:25.658964 506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:35:25.670325 506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:35:25.672462 506 catalog_manager.cc:1259] Loaded cluster ID: f0bf2c2a8bce4c03a9ab121ec9e243f9
I20260810 23:35:25.672732 506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:35:25.677364 506 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 23:35:25.698194 506 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 23:35:25.706434 506 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:35:25.707939 506 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: Loaded TSK: 0
I20260810 23:35:25.709337 506 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:35:25.972066 491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:25.972432 491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:25.972623 491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:25.980038 491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:25.980206 491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:25.980397 491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:35:25.990569 491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:46731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=37739
--tserver_master_addrs=127.9.48.254:33443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:25.991937 491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:25.993371 491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:26.009367 536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:26.010282 540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:26.010810 491 server_base.cc:1034] running on GCE node
W20260810 23:35:26.012832 535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:26.013763 491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:26.015105 491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:26.016382 491 hybrid_clock.cc:648] HybridClock initialized: now 1786404926016339 us; error 56 us; skew 500 ppm
I20260810 23:35:26.016848 491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:26 dist-test-slave-04bb krb5kdc[31962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404926, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:26.020973 491 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:35:26.022617 491 webserver.cc:460] Webserver started at http://127.9.48.193:37739/ using document root <none> and password file <none>
I20260810 23:35:26.023335 491 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:26.023483 491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:26.029323 491 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:35:26.032368 549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:26.033947 491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:35:26.034192 491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "bc9342795a77415eb176245baee56fac"
format_stamp: "Formatted at 2026-08-10 23:35:22 on dist-test-slave-04bb"
server_key: "c3e6c766be2cabe52fddf20c459c86c8"
server_key_iv: "3d65d795585b65f8f6df0fec91ceac53"
server_key_version: "encryptionkey@0"
I20260810 23:35:26.035223 491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:26.055491 491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:26.060353 491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:26.060684 491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:26.061766 491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:26.063133 491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:26.063293 491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:26.063418 491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:26.063512 491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:26.099223 491 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:46731
I20260810 23:35:26.099339 719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:46731 every 8 connection(s)
I20260810 23:35:26.100838 491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:35:26.108130 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 491
I20260810 23:35:26.108950 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:35525
--local_ip_for_outbound_sockets=127.9.48.194
--tserver_master_addrs=127.9.48.254:33443
--webserver_port=40889
--webserver_interface=127.9.48.194
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:35:26 dist-test-slave-04bb krb5kdc[31962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404926, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:26.126830 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:26.103920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:57639 (local address 127.9.48.254:33443)
0810 23:35:26.104664 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:26.104681 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:26.105097 (+ 416us) server_negotiation.cc:407] Connection header received
0810 23:35:26.107168 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:26.107181 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:26.107659 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:26.108134 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:26.109763 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:26.111850 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:26.113462 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:26.114554 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:26.114759 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:26.118710 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:26.118724 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:26.118747 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:26.118799 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:26.122248 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:26.123239 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:26.123246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:26.123251 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:26.123428 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:26.123911 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:26.123915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:26.123917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:26.124397 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:26.124618 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:26.125538 (+ 920us) server_negotiation.cc:299] Negotiation successful
0810 23:35:26.125856 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":271,"threads_started":1}
I20260810 23:35:26.127310 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:26.104022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:57639)
0810 23:35:26.104912 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:26.104959 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:26.106651 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:26.108245 (+ 1594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:26.108256 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:26.108884 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:26.109518 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:26.109535 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:26.112009 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:26.112013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:26.113314 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:26.113326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:26.114853 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:26.114859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:26.115089 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:26.115895 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:26.115921 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:26.118386 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:26.122419 (+ 4033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:26.122432 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:26.122450 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:26.123090 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:26.123541 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:26.123549 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:26.123556 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:26.123800 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:26.124581 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:26.124592 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:26.125014 (+ 422us) client_negotiation.cc:769] Sending connection context
0810 23:35:26.125343 (+ 329us) client_negotiation.cc:240] Negotiation successful
0810 23:35:26.125952 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":398,"threads_started":1}
I20260810 23:35:26.130280 720 heartbeater.cc:347] Connected to a master server at 127.9.48.254:33443
I20260810 23:35:26.130615 720 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:26.131378 720 heartbeater.cc:511] Master 127.9.48.254:33443 requested a full tablet report, sending...
I20260810 23:35:26.133728 430 ts_manager.cc:194] Registered new tserver with Master: bc9342795a77415eb176245baee56fac (127.9.48.193:46731)
I20260810 23:35:26.135414 430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:57639
W20260810 23:35:26.456552 724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:26.456825 724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:26.457013 724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:26.464188 724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:26.464293 724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:26.464495 724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:35:26.474696 724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:35525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=40889
--tserver_master_addrs=127.9.48.254:33443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:26.476253 724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:26.478291 724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:26.492339 732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:26.492390 733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:26.492270 737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:26.493480 724 server_base.cc:1034] running on GCE node
I20260810 23:35:26.494033 724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:26.494917 724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:26.496140 724 hybrid_clock.cc:648] HybridClock initialized: now 1786404926496077 us; error 60 us; skew 500 ppm
I20260810 23:35:26.496474 724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:26 dist-test-slave-04bb krb5kdc[31962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404926, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:26.499805 724 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:35:26.501376 724 webserver.cc:460] Webserver started at http://127.9.48.194:40889/ using document root <none> and password file <none>
I20260810 23:35:26.502075 724 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:26.502242 724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:26.507613 724 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 23:35:26.510483 748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:26.511864 724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:35:26.512001 724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "2a26c78b6a3647c7a9ae32aaea540fc1"
format_stamp: "Formatted at 2026-08-10 23:35:23 on dist-test-slave-04bb"
server_key: "95cc3b97e57ad531d52ea22180fc3420"
server_key_iv: "e9ef9422a60c0af396a45a3471fdd0d3"
server_key_version: "encryptionkey@0"
I20260810 23:35:26.513003 724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:26.531333 724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:26.535763 724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:26.535986 724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:26.536974 724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:26.538233 724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:26.538329 724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:26.538411 724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:26.538467 724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:26.573253 724 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:35525
I20260810 23:35:26.573302 921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:35525 every 8 connection(s)
I20260810 23:35:26.575131 724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:35:26.581604 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 724
I20260810 23:35:26.582458 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:39501
--local_ip_for_outbound_sockets=127.9.48.195
--tserver_master_addrs=127.9.48.254:33443
--webserver_port=38001
--webserver_interface=127.9.48.195
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:35:26 dist-test-slave-04bb krb5kdc[31962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404926, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:26.599995 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:26.577965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:41327 (local address 127.9.48.254:33443)
0810 23:35:26.578140 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:26.578144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:26.579477 (+ 1333us) server_negotiation.cc:407] Connection header received
0810 23:35:26.581543 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:26.581550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:26.581651 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:26.581754 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:26.584213 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:26.585336 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:26.587817 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:26.588511 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:26.588657 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:26.593178 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:26.593188 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:26.593195 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:26.593244 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:26.596271 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:26.597137 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:26.597148 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:26.597150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:26.597284 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:26.597833 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:26.597841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:26.597847 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:26.598278 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:26.598436 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:26.599224 (+ 788us) server_negotiation.cc:299] Negotiation successful
0810 23:35:26.599507 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:35:26.600770 929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:26.578526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:41327)
0810 23:35:26.579299 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:26.579350 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:26.581203 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:26.582089 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:26.582101 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:26.583152 (+ 1051us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:26.583998 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:26.584029 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:26.585516 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:26.585524 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:26.587648 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:26.587668 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:26.588795 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:26.588803 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:26.589145 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:26.589982 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:26.590017 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:26.592958 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:26.596437 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:26.596444 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:26.596456 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:26.596970 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:26.597412 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:26.597417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:26.597421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:26.597616 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:26.598430 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:26.598435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:26.598756 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 23:35:26.599062 (+ 306us) client_negotiation.cc:240] Negotiation successful
0810 23:35:26.599549 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":335,"threads_started":1}
I20260810 23:35:26.602524 924 heartbeater.cc:347] Connected to a master server at 127.9.48.254:33443
I20260810 23:35:26.602936 924 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:26.603677 924 heartbeater.cc:511] Master 127.9.48.254:33443 requested a full tablet report, sending...
I20260810 23:35:26.605204 430 ts_manager.cc:194] Registered new tserver with Master: 2a26c78b6a3647c7a9ae32aaea540fc1 (127.9.48.194:35525)
I20260810 23:35:26.606009 430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:41327
W20260810 23:35:26.931548 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:26.931844 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:26.932021 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:26.939498 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:26.939642 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:26.939869 930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:35:26.949965 930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:39501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=38001
--tserver_master_addrs=127.9.48.254:33443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:26.951301 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:26.952778 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:26.966646 938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:26.966565 943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:26.966617 940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:26.967437 930 server_base.cc:1034] running on GCE node
I20260810 23:35:26.968088 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:26.969134 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:26.970368 930 hybrid_clock.cc:648] HybridClock initialized: now 1786404926970318 us; error 64 us; skew 500 ppm
I20260810 23:35:26.970697 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:26 dist-test-slave-04bb krb5kdc[31962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404926, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:26.974083 930 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:35:26.975755 930 webserver.cc:460] Webserver started at http://127.9.48.195:38001/ using document root <none> and password file <none>
I20260810 23:35:26.976485 930 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:26.976714 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:26.982271 930 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:35:26.985247 953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:26.986675 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:35:26.986868 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "f7b37e3c16c74680add62a67795f6781"
format_stamp: "Formatted at 2026-08-10 23:35:24 on dist-test-slave-04bb"
server_key: "73eaa0960890c6456ebb4ed5f3a03656"
server_key_iv: "ce76dc43567412c91743d3ef02d0a9ab"
server_key_version: "encryptionkey@0"
I20260810 23:35:26.987895 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:27.010691 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:27.015363 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:27.015652 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:27.016736 930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:27.018134 930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:27.018275 930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:27.018405 930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:27.018496 930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:27.053097 930 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:39501
I20260810 23:35:27.053189 1122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:39501 every 8 connection(s)
I20260810 23:35:27.054575 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:35:27.055892 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 930
Aug 10 23:35:27 dist-test-slave-04bb krb5kdc[31962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404926, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:27.072906 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.057058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55400 (local address 127.9.48.254:33443)
0810 23:35:27.057320 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:27.057330 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:27.057791 (+ 461us) server_negotiation.cc:407] Connection header received
0810 23:35:27.057986 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:27.057990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:27.058101 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:27.058246 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:27.060666 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.061433 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.062816 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.063313 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.063458 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.066717 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:27.066722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:27.066725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:27.066754 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:27.069800 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:27.070698 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:27.070702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:27.070704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:27.070836 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:27.071319 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:27.071326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:27.071329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:27.071701 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:27.071989 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:27.072297 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 23:35:27.072518 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 23:35:27.079151 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.058292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:47927 (local address 127.9.48.254:33443)
0810 23:35:27.058793 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:27.058797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:27.061428 (+ 2631us) server_negotiation.cc:407] Connection header received
0810 23:35:27.063293 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:27.063299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:27.063378 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:27.063483 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:27.065671 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.066419 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.068044 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.068825 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.068967 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.073269 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:27.073281 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:27.073286 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:27.073321 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:27.076433 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:27.077194 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:27.077198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:27.077200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:27.077320 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:27.077714 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:27.077718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:27.077720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:27.077967 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:27.078075 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:27.078734 (+ 659us) server_negotiation.cc:299] Negotiation successful
0810 23:35:27.078878 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":297,"threads_started":1}
I20260810 23:35:27.080070 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.058268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:47927)
0810 23:35:27.061254 (+ 2986us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:27.061301 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:27.063048 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:27.063781 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:27.063793 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:27.064605 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:27.065443 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:27.065474 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.066602 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.066610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:27.067898 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:27.067910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.069052 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.069057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:27.069255 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.069870 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:27.069915 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:27.073022 (+ 3107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:27.076611 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:27.076618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:27.076632 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:27.077038 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:27.077421 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:27.077424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:27.077426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:27.077597 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:27.078074 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:27.078079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:27.078405 (+ 326us) client_negotiation.cc:769] Sending connection context
0810 23:35:27.078640 (+ 235us) client_negotiation.cc:240] Negotiation successful
0810 23:35:27.078999 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2682,"thread_start_us":2474,"threads_started":1}
I20260810 23:35:27.081514 1123 heartbeater.cc:347] Connected to a master server at 127.9.48.254:33443
I20260810 23:35:27.081827 1123 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:27.082541 1123 heartbeater.cc:511] Master 127.9.48.254:33443 requested a full tablet report, sending...
I20260810 23:35:27.083936 430 ts_manager.cc:194] Registered new tserver with Master: f7b37e3c16c74680add62a67795f6781 (127.9.48.195:39501)
I20260810 23:35:27.084823 430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:47927
I20260810 23:35:27.088207 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:35:27.095342 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.089164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55408 (local address 127.9.48.254:33443)
0810 23:35:27.089321 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:27.089326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:27.089842 (+ 516us) server_negotiation.cc:407] Connection header received
0810 23:35:27.090045 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:27.090050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:27.090228 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:27.090371 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:35:27.090748 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.091512 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.093349 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.093780 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.093885 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.094355 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:35:27.094852 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:35:27.094961 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:27.095082 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 23:35:27.095152 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:35:27.096195 430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:35:27.106284 430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:35:27.132568 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.118079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:39501 (local address 127.0.0.1:41286)
0810 23:35:27.119035 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:27.119087 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:27.119282 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:27.121704 (+ 2422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:27.121715 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:27.121759 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:27.122325 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:27.122342 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.125413 (+ 3071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.125422 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:27.128609 (+ 3187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:27.128631 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.131195 (+ 2564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.131204 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:27.131510 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.131583 (+ 73us) client_negotiation.cc:769] Sending connection context
0810 23:35:27.131736 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 23:35:27.131996 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":514,"threads_started":1}
I20260810 23:35:27.134339 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.118502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.9.48.195:39501)
0810 23:35:27.120905 (+ 2403us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:27.120918 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:27.120961 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:35:27.121082 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:27.121092 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:27.121411 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:27.122603 (+ 1192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:27.122980 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.125187 (+ 2207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.128887 (+ 3700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.130888 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.133155 (+ 2267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.133525 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:27.133616 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 23:35:27.133854 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2171,"thread_start_us":1950,"threads_started":1}
I20260810 23:35:27.138490 720 heartbeater.cc:503] Master 127.9.48.254:33443 was elected leader, sending a full tablet report...
I20260810 23:35:27.139570 1024 tablet_service.cc:1467] Processing CreateTablet for tablet fa78564879fb4ec084ca8dd620a767f0 (DEFAULT_TABLE table=test-table [id=b02de09a6d0a491fb232dc638280de9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:35:27.141110 1024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa78564879fb4ec084ca8dd620a767f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:27.144752 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.119345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36854 (local address 127.9.48.194:35525)
0810 23:35:27.135761 (+ 16416us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:27.135769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:27.135797 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:35:27.135907 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:27.135914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:27.136142 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:27.136364 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:27.137001 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.138343 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.141898 (+ 3555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.143449 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.143861 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.144066 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:27.144230 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 23:35:27.144407 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16240,"thread_start_us":16095,"threads_started":1}
I20260810 23:35:27.144767 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.118759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:35525 (local address 127.0.0.1:36854)
0810 23:35:27.119967 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:27.119985 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:27.120121 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:27.136339 (+ 16218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:27.136344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:27.136370 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:27.136761 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:27.136774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.138521 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.138528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:27.141721 (+ 3193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:27.141735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.143700 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.143705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:27.144013 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.144054 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 23:35:27.144170 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 23:35:27.144337 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"mutex_wait_us":1684,"thread_start_us":898,"threads_started":1}
I20260810 23:35:27.145267 1148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.119748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.9.48.193:46731)
0810 23:35:27.135062 (+ 15314us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:27.135071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:27.135103 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:35:27.135221 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:27.135229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:27.135507 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:27.135779 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:27.136183 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.138465 (+ 2282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.142007 (+ 3542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.143449 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.144432 (+ 983us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.144690 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:27.144774 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 23:35:27.144977 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14747,"thread_start_us":14957,"threads_started":1}
I20260810 23:35:27.145315 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.119483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46731 (local address 127.0.0.1:57044)
0810 23:35:27.132832 (+ 13349us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:27.132849 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:27.132943 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:27.135788 (+ 2845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:27.135793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:27.135830 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:27.136006 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:27.136016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.138675 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.138681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:27.140599 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:27.140616 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.143651 (+ 3035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.143659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:27.143943 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.143976 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:35:27.144062 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:35:27.144255 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13266,"mutex_wait_us":21,"thread_start_us":15599,"threads_started":1}
I20260810 23:35:27.148518 624 tablet_service.cc:1467] Processing CreateTablet for tablet fa78564879fb4ec084ca8dd620a767f0 (DEFAULT_TABLE table=test-table [id=b02de09a6d0a491fb232dc638280de9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:35:27.148435 818 tablet_service.cc:1467] Processing CreateTablet for tablet fa78564879fb4ec084ca8dd620a767f0 (DEFAULT_TABLE table=test-table [id=b02de09a6d0a491fb232dc638280de9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:35:27.149766 624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa78564879fb4ec084ca8dd620a767f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:27.149845 818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa78564879fb4ec084ca8dd620a767f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:27.161329 1149 tablet_bootstrap.cc:492] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781: Bootstrap starting.
I20260810 23:35:27.162130 1151 tablet_bootstrap.cc:492] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac: Bootstrap starting.
I20260810 23:35:27.164701 1149 tablet_bootstrap.cc:654] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:27.165740 1151 tablet_bootstrap.cc:654] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:27.165832 1152 tablet_bootstrap.cc:492] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1: Bootstrap starting.
I20260810 23:35:27.166025 1149 log.cc:824] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:27.167121 1151 log.cc:824] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:27.169312 1152 tablet_bootstrap.cc:654] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:27.169557 1149 tablet_bootstrap.cc:492] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781: No bootstrap required, opened a new log
I20260810 23:35:27.169996 1149 ts_tablet_manager.cc:1397] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260810 23:35:27.170024 1151 tablet_bootstrap.cc:492] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac: No bootstrap required, opened a new log
I20260810 23:35:27.170295 1151 ts_tablet_manager.cc:1397] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 23:35:27.170583 1152 log.cc:824] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:27.173717 1152 tablet_bootstrap.cc:492] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1: No bootstrap required, opened a new log
I20260810 23:35:27.173985 1152 ts_tablet_manager.cc:1397] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 23:35:27.177742 1151 raft_consensus.cc:348] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } }
I20260810 23:35:27.178122 1151 raft_consensus.cc:374] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:27.178265 1151 raft_consensus.cc:729] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc9342795a77415eb176245baee56fac, State: Initialized, Role: FOLLOWER
I20260810 23:35:27.178879 1151 consensus_queue.cc:260] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } }
I20260810 23:35:27.180701 1151 ts_tablet_manager.cc:1428] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac: Time spent starting tablet: real 0.010s user 0.008s sys 0.004s
I20260810 23:35:27.180783 1149 raft_consensus.cc:348] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } }
I20260810 23:35:27.181264 1149 raft_consensus.cc:374] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:27.181404 1149 raft_consensus.cc:729] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b37e3c16c74680add62a67795f6781, State: Initialized, Role: FOLLOWER
I20260810 23:35:27.182005 1149 consensus_queue.cc:260] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } }
I20260810 23:35:27.185413 1152 raft_consensus.cc:348] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } }
I20260810 23:35:27.186005 1152 raft_consensus.cc:374] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:27.186143 1152 raft_consensus.cc:729] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a26c78b6a3647c7a9ae32aaea540fc1, State: Initialized, Role: FOLLOWER
I20260810 23:35:27.186866 1152 consensus_queue.cc:260] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } }
I20260810 23:35:27.188062 924 heartbeater.cc:503] Master 127.9.48.254:33443 was elected leader, sending a full tablet report...
I20260810 23:35:27.188961 1152 ts_tablet_manager.cc:1428] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260810 23:35:27.189235 1123 heartbeater.cc:503] Master 127.9.48.254:33443 was elected leader, sending a full tablet report...
I20260810 23:35:27.190282 1149 ts_tablet_manager.cc:1428] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781: Time spent starting tablet: real 0.020s user 0.010s sys 0.008s
I20260810 23:35:27.207448 1161 raft_consensus.cc:482] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:35:27.207789 1161 raft_consensus.cc:504] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } }
I20260810 23:35:27.209724 1161 leader_election.cc:288] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7b37e3c16c74680add62a67795f6781 (127.9.48.195:39501), bc9342795a77415eb176245baee56fac (127.9.48.193:46731)
I20260810 23:35:27.219081 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.210086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:59039 (local address 127.9.48.195:39501)
0810 23:35:27.210309 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:27.210314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:27.210971 (+ 657us) server_negotiation.cc:407] Connection header received
0810 23:35:27.211252 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:27.211261 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:27.211352 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:27.211431 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:27.212384 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.213669 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.216506 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.218036 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.218232 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.218309 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:27.218686 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 23:35:27.218817 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:35:27.219201 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.210083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:39501 (local address 127.9.48.194:59039)
0810 23:35:27.210805 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:27.210854 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:27.211034 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:27.211719 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:27.211726 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:27.211759 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:27.212213 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:27.212229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.213800 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.213810 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:27.216308 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:27.216324 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.218276 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.218283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:27.218450 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.218488 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 23:35:27.218663 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 23:35:27.218799 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":45,"thread_start_us":403,"threads_started":1}
I20260810 23:35:27.219400 1148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.211283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:49907 (local address 127.9.48.193:46731)
0810 23:35:27.211501 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:27.211507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:27.211653 (+ 146us) server_negotiation.cc:407] Connection header received
0810 23:35:27.211790 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:27.211799 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:27.211893 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:27.211995 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:27.212671 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.213821 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.216511 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.218041 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.218227 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.218309 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:27.218950 (+ 641us) server_negotiation.cc:299] Negotiation successful
0810 23:35:27.219097 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:35:27.219918 1163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.210897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46731 (local address 127.9.48.194:49907)
0810 23:35:27.211549 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:27.211563 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:27.211668 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:27.212023 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:27.212026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:27.212048 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:27.212439 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:27.212447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.213959 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.213963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:27.216310 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:27.216344 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.218484 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.218489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:27.218735 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.218777 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 23:35:27.219503 (+ 726us) client_negotiation.cc:240] Negotiation successful
0810 23:35:27.219632 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":109,"thread_start_us":316,"threads_started":1}
I20260810 23:35:27.220203 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa78564879fb4ec084ca8dd620a767f0" candidate_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7b37e3c16c74680add62a67795f6781" is_pre_election: true
I20260810 23:35:27.220854 1057 raft_consensus.cc:2454] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a26c78b6a3647c7a9ae32aaea540fc1 in term 0.
I20260810 23:35:27.220976 654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa78564879fb4ec084ca8dd620a767f0" candidate_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc9342795a77415eb176245baee56fac" is_pre_election: true
I20260810 23:35:27.221493 654 raft_consensus.cc:2454] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a26c78b6a3647c7a9ae32aaea540fc1 in term 0.
I20260810 23:35:27.221554 751 leader_election.cc:302] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a26c78b6a3647c7a9ae32aaea540fc1, f7b37e3c16c74680add62a67795f6781; no voters:
I20260810 23:35:27.222034 1161 raft_consensus.cc:2781] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:35:27.222177 1161 raft_consensus.cc:482] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:35:27.222294 1161 raft_consensus.cc:3037] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:27.224653 1161 raft_consensus.cc:504] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } }
I20260810 23:35:27.225258 1161 leader_election.cc:288] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [CANDIDATE]: Term 1 election: Requested vote from peers f7b37e3c16c74680add62a67795f6781 (127.9.48.195:39501), bc9342795a77415eb176245baee56fac (127.9.48.193:46731)
I20260810 23:35:27.225719 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa78564879fb4ec084ca8dd620a767f0" candidate_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7b37e3c16c74680add62a67795f6781"
I20260810 23:35:27.225832 654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa78564879fb4ec084ca8dd620a767f0" candidate_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc9342795a77415eb176245baee56fac"
I20260810 23:35:27.225925 1057 raft_consensus.cc:3037] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:27.226071 654 raft_consensus.cc:3037] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:27.228232 654 raft_consensus.cc:2454] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26c78b6a3647c7a9ae32aaea540fc1 in term 1.
I20260810 23:35:27.228251 1057 raft_consensus.cc:2454] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26c78b6a3647c7a9ae32aaea540fc1 in term 1.
I20260810 23:35:27.228649 751 leader_election.cc:302] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a26c78b6a3647c7a9ae32aaea540fc1, f7b37e3c16c74680add62a67795f6781; no voters:
I20260810 23:35:27.228986 1161 raft_consensus.cc:2781] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:35:27.229434 1161 raft_consensus.cc:686] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 1 LEADER]: Becoming Leader. State: Replica: 2a26c78b6a3647c7a9ae32aaea540fc1, State: Running, Role: LEADER
I20260810 23:35:27.229908 1161 consensus_queue.cc:237] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } } peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } }
I20260810 23:35:27.234791 430 catalog_manager.cc:5495] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a26c78b6a3647c7a9ae32aaea540fc1 (127.9.48.194). New cstate: current_term: 1 leader_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26c78b6a3647c7a9ae32aaea540fc1" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35525 } health_report { overall_health: HEALTHY } } }
I20260810 23:35:27.255462 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.249893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36866 (local address 127.9.48.194:35525)
0810 23:35:27.250098 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:27.250105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:27.250188 (+ 83us) server_negotiation.cc:407] Connection header received
0810 23:35:27.250426 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:27.250430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:27.250507 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:27.250674 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:35:27.251083 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.251808 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.253339 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.253755 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.253893 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.254380 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:35:27.254851 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:35:27.255011 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:27.255070 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 23:35:27.255164 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:35:27.263753 654 raft_consensus.cc:1261] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac [term 1 FOLLOWER]: Refusing update from remote peer 2a26c78b6a3647c7a9ae32aaea540fc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:35:27.263769 1057 raft_consensus.cc:1261] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781 [term 1 FOLLOWER]: Refusing update from remote peer 2a26c78b6a3647c7a9ae32aaea540fc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:35:27.264780 1165 consensus_queue.cc:1035] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc9342795a77415eb176245baee56fac" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:35:27.265195 1161 consensus_queue.cc:1035] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7b37e3c16c74680add62a67795f6781" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 39501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:35:27.277478 1170 mvcc.cc:205] Tried to move back new op lower bound from 7317114582062440448 to 7317114581937815552. Current Snapshot: MvccSnapshot[applied={T|T < 7317114582062440448}]
I20260810 23:35:27.277935 1169 mvcc.cc:205] Tried to move back new op lower bound from 7317114582062440448 to 7317114581937815552. Current Snapshot: MvccSnapshot[applied={T|T < 7317114582062440448}]
I20260810 23:35:27.290310 429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:35:27.290616 429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:35:27.294421 429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: Sending DeleteTablet for 3 replicas of tablet fa78564879fb4ec084ca8dd620a767f0
I20260810 23:35:27.295647 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet fa78564879fb4ec084ca8dd620a767f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:35:27 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:41286
I20260810 23:35:27.295729 624 tablet_service.cc:1514] Processing DeleteTablet for tablet fa78564879fb4ec084ca8dd620a767f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:35:27 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:57044
I20260810 23:35:27.296011 818 tablet_service.cc:1514] Processing DeleteTablet for tablet fa78564879fb4ec084ca8dd620a767f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:35:27 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:36854
I20260810 23:35:27.297161 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 491
I20260810 23:35:27.309540 1184 tablet_replica.cc:330] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac: stopping tablet replica
I20260810 23:35:27.310264 1185 tablet_replica.cc:330] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1: stopping tablet replica
I20260810 23:35:27.311092 1185 raft_consensus.cc:2229] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:35:27.311745 1184 raft_consensus.cc:2229] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:35:27.312465 1184 raft_consensus.cc:2258] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:35:27.312739 1182 tablet_replica.cc:330] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781: stopping tablet replica
I20260810 23:35:27.313135 1182 raft_consensus.cc:2229] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:35:27.313664 1182 raft_consensus.cc:2258] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:35:27.315167 1185 raft_consensus.cc:2258] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:35:27.316026 1182 ts_tablet_manager.cc:1905] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:35:27.317548 1185 ts_tablet_manager.cc:1905] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:35:27.318436 1184 ts_tablet_manager.cc:1905] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:35:27.321350 1148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.299232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.9.48.193:46731)
0810 23:35:27.299473 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:27.299482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:27.299506 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:35:27.299584 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:27.299592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:27.299754 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:27.299877 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:27.302515 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.304290 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.307058 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.308011 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.308166 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.312377 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:27.312395 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:27.312412 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:27.312458 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:27.317000 (+ 4542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:27.317957 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:27.317966 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:27.317969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:27.318136 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:27.318622 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:27.318628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:27.318632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:27.319105 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:27.319446 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:27.320718 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0810 23:35:27.320967 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 23:35:27.322072 715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:27.326795 1185 ts_tablet_manager.cc:1918] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:35:27.327096 1185 log.cc:1196] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/fa78564879fb4ec084ca8dd620a767f0
I20260810 23:35:27.327944 1185 ts_tablet_manager.cc:1939] T fa78564879fb4ec084ca8dd620a767f0 P 2a26c78b6a3647c7a9ae32aaea540fc1: Deleting consensus metadata
I20260810 23:35:27.329710 1182 ts_tablet_manager.cc:1918] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:35:27.329882 1182 log.cc:1196] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/fa78564879fb4ec084ca8dd620a767f0
I20260810 23:35:27.330464 1182 ts_tablet_manager.cc:1939] T fa78564879fb4ec084ca8dd620a767f0 P f7b37e3c16c74680add62a67795f6781: Deleting consensus metadata
I20260810 23:35:27.330502 407 catalog_manager.cc:4841] TS 2a26c78b6a3647c7a9ae32aaea540fc1 (127.9.48.194:35525): tablet fa78564879fb4ec084ca8dd620a767f0 (table test-table [id=b02de09a6d0a491fb232dc638280de9b]) successfully deleted
I20260810 23:35:27.332333 405 catalog_manager.cc:4841] TS f7b37e3c16c74680add62a67795f6781 (127.9.48.195:39501): tablet fa78564879fb4ec084ca8dd620a767f0 (table test-table [id=b02de09a6d0a491fb232dc638280de9b]) successfully deleted
I20260810 23:35:27.434193 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 491
W20260810 23:35:27.460017 405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.48.193:46731 (error 108)
I20260810 23:35:27.460290 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 724
I20260810 23:35:27.462348 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.461852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46731 (local address 127.0.0.1:57074)
0810 23:35:27.462045 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:27.462181 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260810 23:35:27.462643 405 catalog_manager.cc:4584] TS bc9342795a77415eb176245baee56fac (127.9.48.193:46731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa78564879fb4ec084ca8dd620a767f0: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46731: connect: Connection refused (error 111)
I20260810 23:35:27.477599 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.461346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.9.48.194:35525)
0810 23:35:27.461549 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:27.461556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:27.461582 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:35:27.461798 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:27.461803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:27.461871 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:27.461997 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:27.463896 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.464881 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.466795 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.467238 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.467331 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.470846 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:27.470858 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:27.470878 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:27.470916 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:27.474109 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:27.475031 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:27.475041 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:27.475044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:27.475243 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:27.475677 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:27.475681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:27.475684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:27.476133 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:27.476437 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:27.476908 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 23:35:27.477278 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:35:27.478549 915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:27.516225 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.515750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46731 (local address 127.0.0.1:57076)
0810 23:35:27.515985 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:27.516074 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 23:35:27.517338 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.516928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46731 (local address 127.0.0.1:57086)
0810 23:35:27.517105 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:27.517202 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 23:35:27.577926 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.577384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46731 (local address 127.0.0.1:57096)
0810 23:35:27.577621 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:27.577740 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 23:35:27.579124 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.578696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46731 (local address 127.0.0.1:57098)
0810 23:35:27.578858 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:27.578946 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 23:35:27.595616 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 724
I20260810 23:35:27.624593 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 930
I20260810 23:35:27.639204 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.625464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.9.48.195:39501)
0810 23:35:27.625706 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:27.625713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:27.625842 (+ 129us) server_negotiation.cc:407] Connection header received
0810 23:35:27.626074 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:27.626078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:27.626194 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:27.626307 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:27.627742 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.628484 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.630044 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:27.630439 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:27.630533 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:27.633587 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:27.633601 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:27.633618 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:27.633649 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:27.636356 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:27.637255 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:27.637262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:27.637264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:27.637413 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:27.637839 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:27.637842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:27.637844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:27.638185 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:27.638425 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:27.638697 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 23:35:27.638861 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 23:35:27.639954 1118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:27.659747 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.659310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46731 (local address 127.0.0.1:57102)
0810 23:35:27.659506 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:27.659600 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:35:27.660908 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.660475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46731 (local address 127.0.0.1:57118)
0810 23:35:27.660724 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:27.660795 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260810 23:35:27.731218 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 930
I20260810 23:35:27.759507 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 379
I20260810 23:35:27.760901 479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:27.875994 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:27.875012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46731 (local address 127.0.0.1:57126)
0810 23:35:27.875245 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:27.875377 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260810 23:35:27.876006 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 379
I20260810 23:35:27.909020 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:33443
--webserver_interface=127.9.48.254
--webserver_port=36043
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:33443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:35:28.258692 1193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:28.258986 1193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:28.259078 1193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:28.266353 1193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:35:28.266464 1193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:28.266512 1193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:28.266571 1193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:35:28.266616 1193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:35:28.276926 1193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:33443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:33443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=36043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:28.278255 1193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:28.279582 1193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:28.291602 1204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:28.291574 1205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:28.291771 1208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:28.292608 1193 server_base.cc:1034] running on GCE node
I20260810 23:35:28.293437 1193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:28.294795 1193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:28.296005 1193 hybrid_clock.cc:648] HybridClock initialized: now 1786404928295942 us; error 55 us; skew 500 ppm
I20260810 23:35:28.296310 1193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:28 dist-test-slave-04bb krb5kdc[31962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404928, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:28.299664 1193 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:35:28.301357 1193 webserver.cc:460] Webserver started at http://127.9.48.254:36043/ using document root <none> and password file <none>
I20260810 23:35:28.302064 1193 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:28.302204 1193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:28.307747 1193 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:35:28.310698 1223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:28.312021 1193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:35:28.312206 1193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5"
format_stamp: "Formatted at 2026-08-10 23:35:22 on dist-test-slave-04bb"
server_key: "6f19b43ca1a0893c59b79551201a36a4"
server_key_iv: "275bc01dcf2e410a9e4e2021125a1c98"
server_key_version: "encryptionkey@0"
I20260810 23:35:28.313262 1193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:28.357754 1193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:28.362010 1193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:28.362320 1193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:28.382184 1193 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:33443
I20260810 23:35:28.382211 1301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:33443 every 8 connection(s)
I20260810 23:35:28.383711 1193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:35:28.387580 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 1193
I20260810 23:35:28.388445 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:46731
--local_ip_for_outbound_sockets=127.9.48.193
--tserver_master_addrs=127.9.48.254:33443
--webserver_port=37739
--webserver_interface=127.9.48.193
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:35:28.390767 1302 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:35:28.394076 1302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: Bootstrap starting.
I20260810 23:35:28.415846 1302 log.cc:824] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:28.432245 1302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:35:28.432907 1302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5: Bootstrap complete.
I20260810 23:35:28.442215 1302 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } }
I20260810 23:35:28.443179 1302 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2dd8b84b8a4baeb717cb50a8b1a9f5, State: Initialized, Role: FOLLOWER
I20260810 23:35:28.443822 1302 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } }
I20260810 23:35:28.444097 1302 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:35:28.444239 1302 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:35:28.444422 1302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:35:28.448449 1302 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } }
I20260810 23:35:28.449079 1302 leader_election.cc:302] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd2dd8b84b8a4baeb717cb50a8b1a9f5; no voters:
I20260810 23:35:28.449823 1302 leader_election.cc:288] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 23:35:28.450007 1306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:35:28.450805 1306 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [term 3 LEADER]: Becoming Leader. State: Replica: cd2dd8b84b8a4baeb717cb50a8b1a9f5, State: Running, Role: LEADER
I20260810 23:35:28.451558 1306 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } }
I20260810 23:35:28.453029 1302 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:35:28.454219 1310 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd2dd8b84b8a4baeb717cb50a8b1a9f5. Latest consensus state: current_term: 3 leader_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } } }
I20260810 23:35:28.454582 1310 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:28.459065 1317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:35:28.458988 1311 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2dd8b84b8a4baeb717cb50a8b1a9f5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33443 } } }
I20260810 23:35:28.459477 1311 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2dd8b84b8a4baeb717cb50a8b1a9f5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:28.477007 1317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:35:28.479045 1317 catalog_manager.cc:1259] Loaded cluster ID: f0bf2c2a8bce4c03a9ab121ec9e243f9
I20260810 23:35:28.479272 1317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 23:35:28.482045 1317 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
I20260810 23:35:28.487545 1337 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f69b04909ad google::LogMessage::Fail() at ??:0
@ 0x7f69b049388a google::LogMessage::SendToLog() at ??:0
@ 0x7f69b04904de google::LogMessage::Flush() at ??:0
@ 0x7f69b0491a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f69c5ec7512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f69c5ec4976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f69c5f4c610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f69c5f4c439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f69b30dc345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f69b30df91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f69b30df799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f69b30baeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f69b63946db start_thread at ??:0
@ 0x7f69af32a61f clone at ??:0
W20260810 23:35:28.771298 1304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:28.771620 1304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:28.771808 1304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:28.779292 1304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:28.779445 1304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:28.779637 1304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:35:28.789390 1304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:46731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=37739
--tserver_master_addrs=127.9.48.254:33443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:28.790702 1304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:28.792045 1304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:28.806524 1376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:28.806545 1374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:28.806782 1373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:28.807974 1304 server_base.cc:1034] running on GCE node
I20260810 23:35:28.808691 1304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:28.809965 1304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:28.811192 1304 hybrid_clock.cc:648] HybridClock initialized: now 1786404928811109 us; error 80 us; skew 500 ppm
I20260810 23:35:28.811486 1304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:28 dist-test-slave-04bb krb5kdc[31962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404928, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:28.814893 1304 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:35:28.816445 1304 webserver.cc:460] Webserver started at http://127.9.48.193:37739/ using document root <none> and password file <none>
I20260810 23:35:28.817206 1304 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:28.817353 1304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:28.823052 1304 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 23:35:28.826215 1385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:28.827632 1304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:35:28.827843 1304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "bc9342795a77415eb176245baee56fac"
format_stamp: "Formatted at 2026-08-10 23:35:22 on dist-test-slave-04bb"
server_key: "c3e6c766be2cabe52fddf20c459c86c8"
server_key_iv: "3d65d795585b65f8f6df0fec91ceac53"
server_key_version: "encryptionkey@0"
I20260810 23:35:28.828912 1304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:28.876926 1304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:28.883576 1304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:28.883939 1304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:28.885386 1304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:28.888098 1397 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260810 23:35:28.897457 1397 ts_tablet_manager.cc:1852] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260810 23:35:28.898835 1397 ts_tablet_manager.cc:1905] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:35:28.906424 1397 ts_tablet_manager.cc:1918] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:35:28.906634 1397 log.cc:1196] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/fa78564879fb4ec084ca8dd620a767f0
I20260810 23:35:28.907231 1397 ts_tablet_manager.cc:1939] T fa78564879fb4ec084ca8dd620a767f0 P bc9342795a77415eb176245baee56fac: Deleting consensus metadata
I20260810 23:35:28.907665 1304 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260810 23:35:28.907815 1304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.020s user 0.001s sys 0.000s
I20260810 23:35:28.907996 1304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:28.908113 1304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:28.946489 1304 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:46731
I20260810 23:35:28.946540 1560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:46731 every 8 connection(s)
I20260810 23:35:28.947994 1304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:35:28.953213 1564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:28.951156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:46247)
0810 23:35:28.951980 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:28.952237 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":324,"threads_started":1}
W20260810 23:35:28.954061 1561 heartbeater.cc:644] Failed to heartbeat to 127.9.48.254:33443 (0 consecutive failures): Network error: Failed to ping master at 127.9.48.254:33443: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
I20260810 23:35:28.954056 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 1304
I20260810 23:35:28.954815 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:35525
--local_ip_for_outbound_sockets=127.9.48.194
--tserver_master_addrs=127.9.48.254:33443
--webserver_port=40889
--webserver_interface=127.9.48.194
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:35:28.955085 1564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:28.954623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:37045)
0810 23:35:28.954818 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:28.954912 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 23:35:28.956090 1564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:28.955704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:54881)
0810 23:35:28.955867 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:28.955953 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260810 23:35:28.956427 1561 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:35:29.302940 1565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:29.303248 1565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:29.303467 1565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:29.311955 1565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:29.312125 1565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:29.312335 1565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:35:29.322660 1565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:35525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=40889
--tserver_master_addrs=127.9.48.254:33443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:29.324200 1565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:29.325743 1565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:29.339843 1573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:29.339877 1576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:29.339910 1572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:29.341356 1565 server_base.cc:1034] running on GCE node
I20260810 23:35:29.341958 1565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:29.343048 1565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:29.344281 1565 hybrid_clock.cc:648] HybridClock initialized: now 1786404929344235 us; error 52 us; skew 500 ppm
I20260810 23:35:29.344693 1565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:29 dist-test-slave-04bb krb5kdc[31962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404929, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:29.348026 1565 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:35:29.349531 1565 webserver.cc:460] Webserver started at http://127.9.48.194:40889/ using document root <none> and password file <none>
I20260810 23:35:29.350244 1565 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:29.350346 1565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:29.355659 1565 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 23:35:29.358363 1588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:29.359707 1565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:35:29.359894 1565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "2a26c78b6a3647c7a9ae32aaea540fc1"
format_stamp: "Formatted at 2026-08-10 23:35:23 on dist-test-slave-04bb"
server_key: "95cc3b97e57ad531d52ea22180fc3420"
server_key_iv: "e9ef9422a60c0af396a45a3471fdd0d3"
server_key_version: "encryptionkey@0"
I20260810 23:35:29.360970 1565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:29.385558 1565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:29.389703 1565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:29.389997 1565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:29.390973 1565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:29.392375 1565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:29.392508 1565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:29.392660 1565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:29.392725 1565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:29.426508 1565 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:35525
I20260810 23:35:29.426600 1755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:35525 every 8 connection(s)
I20260810 23:35:29.428006 1565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:35:29.433038 1762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:29.431037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:34281)
0810 23:35:29.431772 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:29.431995 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":284,"threads_started":1}
W20260810 23:35:29.433707 1758 heartbeater.cc:644] Failed to heartbeat to 127.9.48.254:33443 (0 consecutive failures): Network error: Failed to ping master at 127.9.48.254:33443: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
I20260810 23:35:29.434528 1762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:29.434209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:37397)
0810 23:35:29.434365 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:29.434429 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 23:35:29.435361 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 1565
I20260810 23:35:29.435456 1762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:29.435036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:52637)
0810 23:35:29.435194 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:29.435288 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:35:29.435684 1758 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:35:29.436107 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:39501
--local_ip_for_outbound_sockets=127.9.48.195
--tserver_master_addrs=127.9.48.254:33443
--webserver_port=38001
--webserver_interface=127.9.48.195
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:35:29.780145 1763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:29.780471 1763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:29.780666 1763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:29.787961 1763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:29.788110 1763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:29.788281 1763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:35:29.798123 1763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:39501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=38001
--tserver_master_addrs=127.9.48.254:33443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:29.799518 1763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:29.800920 1763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:29.814353 1776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:29.814375 1769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:29.814548 1771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:29.816184 1763 server_base.cc:1034] running on GCE node
I20260810 23:35:29.816934 1763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:29.818060 1763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:29.819267 1763 hybrid_clock.cc:648] HybridClock initialized: now 1786404929819200 us; error 59 us; skew 500 ppm
I20260810 23:35:29.819547 1763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:29 dist-test-slave-04bb krb5kdc[31962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404929, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:29.822801 1763 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:35:29.824290 1763 webserver.cc:460] Webserver started at http://127.9.48.195:38001/ using document root <none> and password file <none>
I20260810 23:35:29.825104 1763 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:29.825256 1763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:29.830964 1763 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260810 23:35:29.833909 1784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:29.835379 1763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:35:29.835645 1763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "f7b37e3c16c74680add62a67795f6781"
format_stamp: "Formatted at 2026-08-10 23:35:24 on dist-test-slave-04bb"
server_key: "73eaa0960890c6456ebb4ed5f3a03656"
server_key_iv: "ce76dc43567412c91743d3ef02d0a9ab"
server_key_version: "encryptionkey@0"
I20260810 23:35:29.836766 1763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:29.863204 1763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:29.867959 1763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:29.868287 1763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:29.869496 1763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:29.871034 1763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:29.871174 1763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:29.871323 1763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:29.871424 1763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:29.907763 1763 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:39501
I20260810 23:35:29.907838 1953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:39501 every 8 connection(s)
I20260810 23:35:29.909621 1763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:35:29.914752 1958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:29.912751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:46627)
0810 23:35:29.913525 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:29.913804 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":297,"threads_started":1}
W20260810 23:35:29.915624 1955 heartbeater.cc:644] Failed to heartbeat to 127.9.48.254:33443 (0 consecutive failures): Network error: Failed to ping master at 127.9.48.254:33443: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
I20260810 23:35:29.916455 1958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:29.916121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:47743)
0810 23:35:29.916273 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:29.916345 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 23:35:29.916477 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 1763
I20260810 23:35:29.917564 1958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:29.917136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:44469)
0810 23:35:29.917315 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:29.917417 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 23:35:29.917882 1955 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:35:29.957834 1960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:29.957022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:60911)
0810 23:35:29.957510 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:29.957597 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":240,"threads_started":1}
I20260810 23:35:30.437042 1962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:30.436215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:55055)
0810 23:35:30.436704 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:30.436792 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":209,"threads_started":1}
I20260810 23:35:30.919193 1964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:30.918400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:41423)
0810 23:35:30.918874 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:30.918961 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":230,"threads_started":1}
I20260810 23:35:30.959666 1965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:30.958865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:42615)
0810 23:35:30.959396 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:30.959481 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":299,"threads_started":1}
I20260810 23:35:31.438791 1967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:31.437976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:40805)
0810 23:35:31.438506 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:31.438600 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":291,"threads_started":1}
I20260810 23:35:31.920883 1968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:31.920095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:52589)
0810 23:35:31.920590 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:31.920707 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":207,"threads_started":1}
I20260810 23:35:31.961445 1973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:31.960714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:58389)
0810 23:35:31.961161 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:31.961248 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":219,"threads_started":1}
I20260810 23:35:32.440655 1974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:32.439723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:49475)
0810 23:35:32.440243 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:32.440333 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":269,"threads_started":1}
I20260810 23:35:32.922529 1977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:32.921784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:44849)
0810 23:35:32.922247 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:32.922343 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":243,"threads_started":1}
I20260810 23:35:32.963287 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:32.962442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:35437)
0810 23:35:32.963030 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:32.963121 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":345,"threads_started":1}
I20260810 23:35:33.442322 1982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:33.441554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:54751)
0810 23:35:33.442027 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:33.442110 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":243,"threads_started":1}
I20260810 23:35:33.924245 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:33.923479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:40553)
0810 23:35:33.923978 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:33.924072 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":255,"threads_started":1}
I20260810 23:35:33.965332 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:33.964421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:47069)
0810 23:35:33.965018 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:33.965105 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":307,"threads_started":1}
I20260810 23:35:34.444139 1987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:34.443351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:58515)
0810 23:35:34.443872 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:34.443963 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":263,"threads_started":1}
I20260810 23:35:34.926273 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:34.925418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:48527)
0810 23:35:34.925949 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:34.926037 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":264,"threads_started":1}
I20260810 23:35:34.966990 1989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:34.966270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:41835)
0810 23:35:34.966738 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:34.966828 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":244,"threads_started":1}
I20260810 23:35:35.445935 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:35.445211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:47911)
0810 23:35:35.445664 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:35.445756 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":239,"threads_started":1}
I20260810 23:35:35.928093 1991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:35.927291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:38067)
0810 23:35:35.927803 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:35.927922 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":248,"threads_started":1}
I20260810 23:35:35.968767 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:35.967865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:56757)
0810 23:35:35.968407 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:35.968492 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":238,"threads_started":1}
I20260810 23:35:36.447882 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:36.446845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:52187)
0810 23:35:36.447447 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:36.447556 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":284,"threads_started":1}
I20260810 23:35:36.930189 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:36.929212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:56035)
0810 23:35:36.929707 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:36.929824 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":205,"threads_started":1}
I20260810 23:35:36.970604 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:36.969748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:60009)
0810 23:35:36.970343 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:36.970428 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":288,"threads_started":1}
I20260810 23:35:37.449669 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:37.448955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:56413)
0810 23:35:37.449429 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:37.449508 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":248,"threads_started":1}
I20260810 23:35:37.931960 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:37.931243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:56591)
0810 23:35:37.931705 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:37.931787 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":237,"threads_started":1}
I20260810 23:35:37.972509 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:37.971615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:39513)
0810 23:35:37.972180 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:37.972314 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":284,"threads_started":1}
I20260810 23:35:38.451426 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:38.450663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:55697)
0810 23:35:38.451162 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:38.451248 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":248,"threads_started":1}
I20260810 23:35:38.934201 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:38.933184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:59553)
0810 23:35:38.933739 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:38.933940 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":234,"threads_started":1}
I20260810 23:35:38.974416 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:38.973641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:45029)
0810 23:35:38.974131 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:38.974220 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":251,"threads_started":1}
I20260810 23:35:39.453130 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:39.452373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:40277)
0810 23:35:39.452877 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:39.452961 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":290,"threads_started":1}
I20260810 23:35:39.935945 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:39.935183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:42213)
0810 23:35:39.935673 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:39.935771 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":270,"threads_started":1}
I20260810 23:35:39.976191 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:39.975426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:52681)
0810 23:35:39.975903 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:39.975985 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":262,"threads_started":1}
I20260810 23:35:40.454766 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:40.454028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:47315)
0810 23:35:40.454484 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:40.454582 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":233,"threads_started":1}
I20260810 23:35:40.937824 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:40.937008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:33335)
0810 23:35:40.937524 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:40.937624 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":286,"threads_started":1}
I20260810 23:35:40.978142 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:40.977240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:52637)
0810 23:35:40.977822 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:40.977926 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":316,"threads_started":1}
I20260810 23:35:41.456722 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:41.455846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:35205)
0810 23:35:41.456410 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:41.456496 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":310,"threads_started":1}
I20260810 23:35:41.941169 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:41.939690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:50771)
0810 23:35:41.940928 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:41.941008 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1112,"thread_start_us":983,"threads_started":1}
I20260810 23:35:41.980130 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:41.979256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:40963)
0810 23:35:41.979792 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:41.979915 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":271,"threads_started":1}
I20260810 23:35:42.458492 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:42.457692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:57111)
0810 23:35:42.458174 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:42.458298 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":242,"threads_started":1}
I20260810 23:35:42.942890 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:42.942070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:54839)
0810 23:35:42.942591 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:42.942693 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":255,"threads_started":1}
I20260810 23:35:42.981942 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:42.981153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:51351)
0810 23:35:42.981651 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:42.981741 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":232,"threads_started":1}
I20260810 23:35:43.460425 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:43.459556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:40253)
0810 23:35:43.460123 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:43.460217 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":316,"threads_started":1}
I20260810 23:35:43.944489 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:43.943770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:55325)
0810 23:35:43.944217 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:43.944297 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":222,"threads_started":1}
I20260810 23:35:43.984052 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:43.983219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.193:32799)
0810 23:35:43.983749 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:43.983847 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":247,"threads_started":1}
I20260810 23:35:44.462762 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:44.461864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.194:49003)
0810 23:35:44.462415 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:44.462545 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":296,"threads_started":1}
I20260810 23:35:44.924291 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 1304
I20260810 23:35:44.946559 2035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:44.945724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33443 (local address 127.9.48.195:40959)
0810 23:35:44.946192 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:44.946303 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:33443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260810 23:35:44.949324 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:44.925672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.9.48.193:46731)
0810 23:35:44.926361 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:44.926377 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:44.926418 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:35:44.928426 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:44.928436 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:44.928965 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:44.929343 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:44.930933 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:44.932178 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:44.933709 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:44.934400 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:44.934549 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:44.942258 (+ 7709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:44.942267 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:44.942284 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:44.942333 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:44.945581 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:44.946462 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:44.946472 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:44.946482 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:44.946724 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:44.947131 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:44.947135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:44.947137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:44.947766 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:44.948042 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:44.948517 (+ 475us) server_negotiation.cc:299] Negotiation successful
0810 23:35:44.949017 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":308,"threads_started":1}
I20260810 23:35:44.950745 1553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:45.037367 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 1304
I20260810 23:35:45.068346 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 1565
I20260810 23:35:45.086305 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:45.069454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.9.48.194:35525)
0810 23:35:45.070105 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:45.070120 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:45.070160 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:35:45.071781 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:45.071790 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:45.072241 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:45.072615 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:45.074015 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:45.075255 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:45.076609 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:45.077308 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:45.077446 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:45.080002 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:45.080010 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:45.080025 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:45.080067 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:45.083183 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:45.083922 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:45.083929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:45.083936 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:45.084101 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:45.084461 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:45.084465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:45.084467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:45.085038 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:45.085295 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:45.085659 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 23:35:45.086007 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":342,"threads_started":1}
I20260810 23:35:45.087733 1749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:45.173920 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 1565
I20260810 23:35:45.200857 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 1763
I20260810 23:35:45.219336 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:45.201902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59850 (local address 127.9.48.195:39501)
0810 23:35:45.202532 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:45.202546 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:45.202598 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:35:45.204259 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:45.204269 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:45.204747 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:45.205112 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:45.206662 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:45.207917 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:45.209388 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:45.210066 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:45.210271 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:45.213076 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:45.213084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:45.213098 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:45.213141 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:45.215934 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:45.216691 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:45.216698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:45.216705 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:45.216872 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:45.217296 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:45.217300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:45.217307 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:45.217816 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:45.218159 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:45.218605 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 23:35:45.219015 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":304,"threads_started":1}
I20260810 23:35:45.221011 1946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:45.308144 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 1763
2026-08-10T23:35:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25076 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:35:45 dist-test-slave-04bb krb5kdc[2048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:35:45 dist-test-slave-04bb krb5kdc[2048](info): set up 2 sockets
Aug 10 23:35:45 dist-test-slave-04bb krb5kdc[2048](info): commencing operation
krb5kdc: starting...
W20260810 23:35:46.670930 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.310s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:35:46 dist-test-slave-04bb krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:35:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:35:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:35:46.996366 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:44525
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:44525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:35:47.349920 2064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:47.350198 2064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:47.350291 2064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:47.357409 2064 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:35:47.357503 2064 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:35:47.357542 2064 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:35:47.357589 2064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:35:47.357645 2064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:47.357712 2064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:47.357757 2064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:35:47.357805 2064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:35:47.367664 2064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:44525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:44525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:47.368917 2064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:47.370216 2064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:47.382587 2074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:47.382654 2076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:47.382619 2079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:47.383409 2064 server_base.cc:1034] running on GCE node
I20260810 23:35:47.384286 2064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:47.385802 2064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:47.387063 2064 hybrid_clock.cc:648] HybridClock initialized: now 1786404947386990 us; error 49 us; skew 500 ppm
I20260810 23:35:47.387416 2064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:47 dist-test-slave-04bb krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404947, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:47.390731 2064 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:35:47.392158 2064 webserver.cc:460] Webserver started at http://127.9.48.254:44461/ using document root <none> and password file <none>
I20260810 23:35:47.392976 2064 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:47.393115 2064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:47.393462 2064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:35:47.396786 2064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "184559be7e544fbeb0b19ac9f7e24f75"
format_stamp: "Formatted at 2026-08-10 23:35:47 on dist-test-slave-04bb"
server_key: "a01df406fde201552a333836f869d2bb"
server_key_iv: "5afb06f08cd3e98ca3ba0f97da81e99b"
server_key_version: "encryptionkey@0"
I20260810 23:35:47.397627 2064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "184559be7e544fbeb0b19ac9f7e24f75"
format_stamp: "Formatted at 2026-08-10 23:35:47 on dist-test-slave-04bb"
server_key: "a01df406fde201552a333836f869d2bb"
server_key_iv: "5afb06f08cd3e98ca3ba0f97da81e99b"
server_key_version: "encryptionkey@0"
I20260810 23:35:47.402837 2064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:35:47.406347 2090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:47.407721 2064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:35:47.407931 2064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "184559be7e544fbeb0b19ac9f7e24f75"
format_stamp: "Formatted at 2026-08-10 23:35:47 on dist-test-slave-04bb"
server_key: "a01df406fde201552a333836f869d2bb"
server_key_iv: "5afb06f08cd3e98ca3ba0f97da81e99b"
server_key_version: "encryptionkey@0"
I20260810 23:35:47.408157 2064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:47.440995 2064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:47.445662 2064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:47.446048 2064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:47.465801 2064 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:44525
I20260810 23:35:47.465821 2168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:44525 every 8 connection(s)
I20260810 23:35:47.467257 2064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:35:47.468743 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 2064
I20260810 23:35:47.468994 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:35:47.469914 9411 external_mini_cluster.cc:1428] Setting key 8a37de2cd7c82b7f0019121cd243f891
I20260810 23:35:47.472429 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:35:47 dist-test-slave-04bb krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:47.486085 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Bootstrap starting.
I20260810 23:35:47.489715 2169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:47.490844 2169 log.cc:824] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:47.494400 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: No bootstrap required, opened a new log
I20260810 23:35:47.495306 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:47.472125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.9.48.254:44525)
0810 23:35:47.472976 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:47.472992 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:47.473044 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:35:47.475402 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:47.475432 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:47.475964 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:47.476424 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:47.477919 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:47.480029 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:47.481456 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:47.482548 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:47.482735 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:47.487341 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:47.487350 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:47.487364 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:47.487406 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:47.490789 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:47.491813 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:47.491829 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:47.491838 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:47.492055 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:47.492595 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:47.492602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:47.492605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:47.493121 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:47.493366 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:47.493870 (+ 504us) server_negotiation.cc:299] Negotiation successful
0810 23:35:47.494230 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":384,"threads_started":1}
I20260810 23:35:47.502251 2169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } }
I20260810 23:35:47.502664 2169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:47.502794 2169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184559be7e544fbeb0b19ac9f7e24f75, State: Initialized, Role: FOLLOWER
I20260810 23:35:47.503355 2169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } }
I20260810 23:35:47.503628 2169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:35:47.503787 2169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:35:47.503994 2169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:47.506381 2169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } }
I20260810 23:35:47.506886 2169 leader_election.cc:302] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 184559be7e544fbeb0b19ac9f7e24f75; no voters:
I20260810 23:35:47.507530 2169 leader_election.cc:288] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:35:47.507686 2179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:35:47.508422 2179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 1 LEADER]: Becoming Leader. State: Replica: 184559be7e544fbeb0b19ac9f7e24f75, State: Running, Role: LEADER
I20260810 23:35:47.509099 2179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } }
I20260810 23:35:47.509714 2169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:35:47.512293 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 184559be7e544fbeb0b19ac9f7e24f75. Latest consensus state: current_term: 1 leader_uuid: "184559be7e544fbeb0b19ac9f7e24f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } } }
I20260810 23:35:47.513031 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:47.512332 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "184559be7e544fbeb0b19ac9f7e24f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } } }
I20260810 23:35:47.513303 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [sys.catalog]: This master's current role is: LEADER
I20260810 23:35:47.515389 2190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:35:47.521106 2190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:35:47.531117 2190 catalog_manager.cc:1347] Generated new cluster ID: 1ffbf61718644dabbf84140370f53026
I20260810 23:35:47.531319 2190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:35:47.542713 2190 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:35:47.544129 2190 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:35:47.551496 2190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Generated new TSK 0
I20260810 23:35:47.552654 2190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:35:47.628697 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:44525
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:35:47.986006 2212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:47.986349 2212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:47.986528 2212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:47.993955 2212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:47.994120 2212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:47.994306 2212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:35:48.004405 2212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:44525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:48.005849 2212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:48.007150 2212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:48.021183 2226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:48.021416 2222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:48.021426 2224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:48.022480 2212 server_base.cc:1034] running on GCE node
I20260810 23:35:48.023136 2212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:48.024077 2212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:48.025321 2212 hybrid_clock.cc:648] HybridClock initialized: now 1786404948025272 us; error 52 us; skew 500 ppm
I20260810 23:35:48.025643 2212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:48 dist-test-slave-04bb krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404948, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:48.028877 2212 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:35:48.030253 2212 webserver.cc:460] Webserver started at http://127.9.48.193:35867/ using document root <none> and password file <none>
I20260810 23:35:48.030951 2212 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:48.031108 2212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:48.031424 2212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:35:48.034373 2212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "837e557a1dde466b948d188ee833077b"
format_stamp: "Formatted at 2026-08-10 23:35:48 on dist-test-slave-04bb"
server_key: "f4037bb04a904f875770deaa19634779"
server_key_iv: "96d6c1e128935fec4fcb3519ba6494e2"
server_key_version: "encryptionkey@0"
I20260810 23:35:48.035086 2212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "837e557a1dde466b948d188ee833077b"
format_stamp: "Formatted at 2026-08-10 23:35:48 on dist-test-slave-04bb"
server_key: "f4037bb04a904f875770deaa19634779"
server_key_iv: "96d6c1e128935fec4fcb3519ba6494e2"
server_key_version: "encryptionkey@0"
I20260810 23:35:48.040180 2212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 23:35:48.043951 2236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:48.045396 2212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:35:48.045588 2212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "837e557a1dde466b948d188ee833077b"
format_stamp: "Formatted at 2026-08-10 23:35:48 on dist-test-slave-04bb"
server_key: "f4037bb04a904f875770deaa19634779"
server_key_iv: "96d6c1e128935fec4fcb3519ba6494e2"
server_key_version: "encryptionkey@0"
I20260810 23:35:48.045786 2212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:48.060635 2212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:48.064903 2212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:48.065208 2212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:48.066195 2212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:48.067579 2212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:48.067687 2212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:48.067859 2212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:48.067945 2212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:48.104100 2212 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:46583
I20260810 23:35:48.104154 2400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:46583 every 8 connection(s)
I20260810 23:35:48.105482 2212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:35:48.113018 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 2212
I20260810 23:35:48.113278 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:35:48.114133 9411 external_mini_cluster.cc:1428] Setting key de29519a60ba65ad7d5af48033496d53
Aug 10 23:35:48 dist-test-slave-04bb krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404948, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:48.127486 2408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:48.108257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:57099 (local address 127.9.48.254:44525)
0810 23:35:48.108767 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:48.108774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:48.109768 (+ 994us) server_negotiation.cc:407] Connection header received
0810 23:35:48.111664 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:48.111670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:48.111784 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:48.111907 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:48.114045 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:48.114949 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:48.116946 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:48.117343 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:48.117447 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:48.121715 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:48.121719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:48.121723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:48.121754 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:48.124295 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:48.125132 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:48.125137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:48.125139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:48.125261 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:48.125733 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:48.125737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:48.125739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:48.125990 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:48.126151 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:48.126863 (+ 712us) server_negotiation.cc:299] Negotiation successful
0810 23:35:48.127096 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":275,"threads_started":1}
I20260810 23:35:48.128453 2411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:48.108847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:57099)
0810 23:35:48.109607 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:48.109647 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:48.111388 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:48.112161 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:48.112169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:48.112907 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:48.113801 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:48.113834 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:48.115141 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:48.115145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:48.116796 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:48.116813 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:48.117585 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:48.117595 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:48.117953 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:48.118711 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:48.118746 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:48.121486 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:48.124460 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:48.124469 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:48.124486 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:48.124991 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:48.125390 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:48.125396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:48.125399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:48.125600 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:48.126126 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:48.126134 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:48.126477 (+ 343us) client_negotiation.cc:769] Sending connection context
0810 23:35:48.126749 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 23:35:48.127279 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":281,"threads_started":1}
I20260810 23:35:48.130198 2401 heartbeater.cc:347] Connected to a master server at 127.9.48.254:44525
I20260810 23:35:48.130533 2401 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:48.131423 2401 heartbeater.cc:511] Master 127.9.48.254:44525 requested a full tablet report, sending...
I20260810 23:35:48.133966 2114 ts_manager.cc:194] Registered new tserver with Master: 837e557a1dde466b948d188ee833077b (127.9.48.193:46583)
I20260810 23:35:48.136169 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:57099
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:35:48.187971 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:44525
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:35:48.538805 2417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:48.539088 2417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:48.539194 2417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:48.546571 2417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:48.546676 2417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:48.546793 2417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:35:48.557497 2417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:44525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:48.559046 2417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:48.560410 2417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:48.573807 2423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:48.573947 2424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:48.573873 2427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:48.574988 2417 server_base.cc:1034] running on GCE node
I20260810 23:35:48.575593 2417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:48.576503 2417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:48.577786 2417 hybrid_clock.cc:648] HybridClock initialized: now 1786404948577734 us; error 44 us; skew 500 ppm
I20260810 23:35:48.578120 2417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:48 dist-test-slave-04bb krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404948, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:48.581427 2417 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:35:48.582938 2417 webserver.cc:460] Webserver started at http://127.9.48.194:37693/ using document root <none> and password file <none>
I20260810 23:35:48.583652 2417 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:48.583807 2417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:48.584131 2417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:35:48.587139 2417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "fa970edeeef3483e981d005986bd0951"
format_stamp: "Formatted at 2026-08-10 23:35:48 on dist-test-slave-04bb"
server_key: "ced851f212b35e5de7496a884cfcf256"
server_key_iv: "75693b5aa77aabb8ec46aa4bc54e162e"
server_key_version: "encryptionkey@0"
I20260810 23:35:48.587955 2417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "fa970edeeef3483e981d005986bd0951"
format_stamp: "Formatted at 2026-08-10 23:35:48 on dist-test-slave-04bb"
server_key: "ced851f212b35e5de7496a884cfcf256"
server_key_iv: "75693b5aa77aabb8ec46aa4bc54e162e"
server_key_version: "encryptionkey@0"
I20260810 23:35:48.593228 2417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260810 23:35:48.596767 2435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:48.598141 2417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:35:48.598330 2417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "fa970edeeef3483e981d005986bd0951"
format_stamp: "Formatted at 2026-08-10 23:35:48 on dist-test-slave-04bb"
server_key: "ced851f212b35e5de7496a884cfcf256"
server_key_iv: "75693b5aa77aabb8ec46aa4bc54e162e"
server_key_version: "encryptionkey@0"
I20260810 23:35:48.598532 2417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:48.636106 2417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:48.640494 2417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:48.640873 2417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:48.641904 2417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:48.643289 2417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:48.643396 2417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:48.643566 2417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:48.643640 2417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:48.677886 2417 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:44837
I20260810 23:35:48.677980 2602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:44837 every 8 connection(s)
I20260810 23:35:48.679251 2417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:35:48.680763 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 2417
I20260810 23:35:48.680980 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:35:48.681471 9411 external_mini_cluster.cc:1428] Setting key e4f27bd838997477cd6340a266d6d87c
Aug 10 23:35:48 dist-test-slave-04bb krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404948, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:48.700764 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:48.682048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:51891 (local address 127.9.48.254:44525)
0810 23:35:48.682526 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:48.682531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:48.684052 (+ 1521us) server_negotiation.cc:407] Connection header received
0810 23:35:48.685721 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:48.685727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:48.685839 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:48.685961 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:48.687490 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:48.688234 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:48.689856 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:48.690279 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:48.690419 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:48.693959 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:48.693964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:48.693967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:48.693999 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:48.697264 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:48.698016 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:48.698025 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:48.698031 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:48.698202 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:48.698628 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:48.698636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:48.698642 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:48.699047 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:48.699205 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:48.699939 (+ 734us) server_negotiation.cc:299] Negotiation successful
0810 23:35:48.700224 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":224,"threads_started":1}
I20260810 23:35:48.701481 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:48.682660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:51891)
0810 23:35:48.683584 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:48.683638 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:48.685448 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:48.686209 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:48.686218 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:48.686734 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:48.687311 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:48.687330 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:48.688409 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:48.688413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:48.689700 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:48.689713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:48.690521 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:48.690528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:48.690755 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:48.691463 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:48.691492 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:48.693707 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:48.697421 (+ 3714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:48.697428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:48.697442 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:48.697872 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:48.698315 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:48.698319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:48.698321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:48.698505 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:48.699170 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:48.699176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:48.699497 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 23:35:48.699753 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 23:35:48.700149 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":380,"threads_started":1}
I20260810 23:35:48.703083 2603 heartbeater.cc:347] Connected to a master server at 127.9.48.254:44525
I20260810 23:35:48.703485 2603 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:48.704406 2603 heartbeater.cc:511] Master 127.9.48.254:44525 requested a full tablet report, sending...
I20260810 23:35:48.706110 2114 ts_manager.cc:194] Registered new tserver with Master: fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837)
I20260810 23:35:48.706972 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:51891
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:35:48.755584 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:44525
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:35:49.120319 2613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:35:49.120651 2613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:35:49.120858 2613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:35:49.128087 2613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:35:49.128206 2613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:35:49.128438 2613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:35:49.138065 2613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:44525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:35:49.139393 2613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:35:49.140168 2401 heartbeater.cc:503] Master 127.9.48.254:44525 was elected leader, sending a full tablet report...
I20260810 23:35:49.140976 2613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:35:49.155004 2623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:49.154963 2626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:35:49.155043 2621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:35:49.155915 2613 server_base.cc:1034] running on GCE node
I20260810 23:35:49.156635 2613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:35:49.157619 2613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:35:49.158865 2613 hybrid_clock.cc:648] HybridClock initialized: now 1786404949158825 us; error 52 us; skew 500 ppm
I20260810 23:35:49.159209 2613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:35:49 dist-test-slave-04bb krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404949, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:35:49.163048 2613 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:35:49.164453 2613 webserver.cc:460] Webserver started at http://127.9.48.195:33069/ using document root <none> and password file <none>
I20260810 23:35:49.165203 2613 fs_manager.cc:359] Metadata directory not provided
I20260810 23:35:49.165308 2613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:35:49.165581 2613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:35:49.168644 2613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "229f01207aec449cb3948ad9596890a6"
format_stamp: "Formatted at 2026-08-10 23:35:49 on dist-test-slave-04bb"
server_key: "ace0af1a573b449577c7561794a865c6"
server_key_iv: "e76d9bf63d249ce9f6aeb6cc88ce945d"
server_key_version: "encryptionkey@0"
I20260810 23:35:49.169564 2613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "229f01207aec449cb3948ad9596890a6"
format_stamp: "Formatted at 2026-08-10 23:35:49 on dist-test-slave-04bb"
server_key: "ace0af1a573b449577c7561794a865c6"
server_key_iv: "e76d9bf63d249ce9f6aeb6cc88ce945d"
server_key_version: "encryptionkey@0"
I20260810 23:35:49.174752 2613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:35:49.178689 2633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:49.180143 2613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:35:49.180315 2613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "229f01207aec449cb3948ad9596890a6"
format_stamp: "Formatted at 2026-08-10 23:35:49 on dist-test-slave-04bb"
server_key: "ace0af1a573b449577c7561794a865c6"
server_key_iv: "e76d9bf63d249ce9f6aeb6cc88ce945d"
server_key_version: "encryptionkey@0"
I20260810 23:35:49.180496 2613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:35:49.199132 2613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:35:49.203579 2613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:35:49.203830 2613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:35:49.204779 2613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:35:49.206103 2613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:35:49.206205 2613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:49.206290 2613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:35:49.206334 2613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:35:49.241946 2613 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:44079
I20260810 23:35:49.242036 2827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:44079 every 8 connection(s)
I20260810 23:35:49.243274 2613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:35:49.251881 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 2613
I20260810 23:35:49.252092 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:35:49.252655 9411 external_mini_cluster.cc:1428] Setting key 86ca85307d116ebf5ded7c3dbe824fec
Aug 10 23:35:49 dist-test-slave-04bb krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404949, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:35:49.264144 2832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.245971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:55719 (local address 127.9.48.254:44525)
0810 23:35:49.246442 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:49.246447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:49.247270 (+ 823us) server_negotiation.cc:407] Connection header received
0810 23:35:49.249448 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:49.249453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:49.249573 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:49.249707 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:49.251594 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.252600 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.254368 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.254873 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.255016 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.258621 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:49.258626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:49.258630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:49.258661 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:49.261430 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:49.262100 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:49.262104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:49.262106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:49.262226 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:49.262638 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:49.262641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:49.262643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:49.262893 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:49.263022 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:49.263701 (+ 679us) server_negotiation.cc:299] Negotiation successful
0810 23:35:49.263878 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":229,"threads_started":1}
I20260810 23:35:49.265190 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.246342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:55719)
0810 23:35:49.247024 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:49.247094 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:49.249140 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:49.249988 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:49.249997 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:49.250627 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:35:49.251197 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:49.251215 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.252769 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.252774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:49.254210 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:49.254226 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.255114 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.255121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:49.255360 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.256077 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:35:49.256106 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:35:49.258427 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:35:49.261558 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:49.261565 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:49.261577 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:49.261999 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:49.262347 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:35:49.262351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:35:49.262354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:35:49.262536 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:35:49.263026 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:35:49.263031 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:35:49.263370 (+ 339us) client_negotiation.cc:769] Sending connection context
0810 23:35:49.263603 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 23:35:49.264000 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":266,"threads_started":1}
I20260810 23:35:49.266710 2828 heartbeater.cc:347] Connected to a master server at 127.9.48.254:44525
I20260810 23:35:49.267081 2828 heartbeater.cc:464] Registering TS with master...
I20260810 23:35:49.268034 2828 heartbeater.cc:511] Master 127.9.48.254:44525 requested a full tablet report, sending...
I20260810 23:35:49.269572 2114 ts_manager.cc:194] Registered new tserver with Master: 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079)
I20260810 23:35:49.270375 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:55719
I20260810 23:35:49.280460 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:35:49.298843 2832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.285321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.9.48.254:44525)
0810 23:35:49.285541 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:49.285551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:49.286062 (+ 511us) server_negotiation.cc:407] Connection header received
0810 23:35:49.286247 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:49.286251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:49.286317 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:49.286400 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:49.287958 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.288731 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.290278 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.290668 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.290765 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.293440 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:49.293445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:49.293448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:49.293478 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:49.296241 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:49.297001 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:49.297006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:49.297009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:49.297125 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:49.297508 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:49.297511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:49.297513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:49.297831 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:49.298021 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:49.298383 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 23:35:49.298544 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:35:49.303011 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:35:49.309947 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:35:49.346611 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.322215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.9.48.193:46583)
0810 23:35:49.335288 (+ 13073us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:49.335313 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:49.336597 (+ 1284us) server_negotiation.cc:407] Connection header received
0810 23:35:49.336717 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:49.336727 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:49.337026 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:49.337228 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:49.338378 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.340577 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.343721 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.345310 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.345464 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.345720 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:49.346070 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 23:35:49.346244 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12813,"thread_start_us":13113,"threads_started":1}
I20260810 23:35:49.346771 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.322144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46583 (local address 127.0.0.1:60300)
0810 23:35:49.336121 (+ 13977us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:49.336172 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:49.336366 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:49.337603 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:49.337616 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:49.337659 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:49.338132 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:49.338149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.340818 (+ 2669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.340826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:49.343528 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:49.343546 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.345533 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.345539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:49.345768 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.345850 (+ 82us) client_negotiation.cc:769] Sending connection context
0810 23:35:49.346006 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 23:35:49.346250 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13736,"mutex_wait_us":1,"thread_start_us":13594,"threads_started":1}
I20260810 23:35:49.350242 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.324182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:44837 (local address 127.0.0.1:45002)
0810 23:35:49.339557 (+ 15375us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:49.339581 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:49.339747 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:49.342397 (+ 2650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:49.342403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:49.342436 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:49.342915 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:49.342928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.344814 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.344820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:49.347134 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:49.347170 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.349455 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.349462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:49.349638 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.349673 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:35:49.349785 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 23:35:49.349906 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15211,"thread_start_us":15668,"threads_started":1}
I20260810 23:35:49.351045 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.324483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45002 (local address 127.9.48.194:44837)
0810 23:35:49.341488 (+ 17005us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:49.341499 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:49.341534 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:35:49.341639 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:49.341648 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:49.341932 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:49.342126 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:49.343125 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.344636 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.347339 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.348952 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.349081 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.349316 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:49.350523 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0810 23:35:49.350715 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16819,"thread_start_us":16643,"threads_started":1}
I20260810 23:35:49.351152 2309 tablet_service.cc:1467] Processing CreateTablet for tablet 32b11fedf70849ac86b0ea8ae9601e57 (DEFAULT_TABLE table=test-table [id=3a1dbe5f0cdb45b08602eb21b8bfc4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:35:49.352773 2309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b11fedf70849ac86b0ea8ae9601e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:49.352934 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.340813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:44079 (local address 127.0.0.1:38792)
0810 23:35:49.341784 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:49.341801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:49.341917 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:49.343295 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:49.343300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:49.343325 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:49.343803 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:49.343816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.345886 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.345890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:49.348545 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:49.348562 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.350733 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.350739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:49.350912 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.350943 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 23:35:49.351039 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 23:35:49.352610 (+ 1571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":1193,"threads_started":1}
I20260810 23:35:49.353677 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.337042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.9.48.195:44079)
0810 23:35:49.342709 (+ 5667us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:49.342719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:49.342758 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:35:49.342861 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:49.342870 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:49.343100 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:49.343306 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:49.344004 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.345670 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.348754 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.350225 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.350362 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.350602 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:49.353177 (+ 2575us) server_negotiation.cc:299] Negotiation successful
0810 23:35:49.353343 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5468,"thread_start_us":5277,"threads_started":1}
I20260810 23:35:49.354592 2501 tablet_service.cc:1467] Processing CreateTablet for tablet 32b11fedf70849ac86b0ea8ae9601e57 (DEFAULT_TABLE table=test-table [id=3a1dbe5f0cdb45b08602eb21b8bfc4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:35:49.355682 2501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b11fedf70849ac86b0ea8ae9601e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:49.361647 2713 tablet_service.cc:1467] Processing CreateTablet for tablet 32b11fedf70849ac86b0ea8ae9601e57 (DEFAULT_TABLE table=test-table [id=3a1dbe5f0cdb45b08602eb21b8bfc4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:35:49.363139 2713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b11fedf70849ac86b0ea8ae9601e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:35:49.364957 2858 tablet_bootstrap.cc:492] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b: Bootstrap starting.
I20260810 23:35:49.368150 2858 tablet_bootstrap.cc:654] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:49.369287 2859 tablet_bootstrap.cc:492] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951: Bootstrap starting.
I20260810 23:35:49.369954 2858 log.cc:824] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:49.373170 2859 tablet_bootstrap.cc:654] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:49.373972 2858 tablet_bootstrap.cc:492] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b: No bootstrap required, opened a new log
I20260810 23:35:49.374279 2858 ts_tablet_manager.cc:1397] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260810 23:35:49.374473 2859 log.cc:824] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:49.376997 2862 tablet_bootstrap.cc:492] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6: Bootstrap starting.
I20260810 23:35:49.377725 2859 tablet_bootstrap.cc:492] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951: No bootstrap required, opened a new log
I20260810 23:35:49.378006 2859 ts_tablet_manager.cc:1397] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 23:35:49.379953 2862 tablet_bootstrap.cc:654] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6: Neither blocks nor log segments found. Creating new log.
I20260810 23:35:49.381127 2862 log.cc:824] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:35:49.384317 2862 tablet_bootstrap.cc:492] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6: No bootstrap required, opened a new log
I20260810 23:35:49.384644 2862 ts_tablet_manager.cc:1397] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260810 23:35:49.386453 2858 raft_consensus.cc:348] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } } peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } }
I20260810 23:35:49.386914 2858 raft_consensus.cc:374] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:49.387013 2858 raft_consensus.cc:729] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 837e557a1dde466b948d188ee833077b, State: Initialized, Role: FOLLOWER
I20260810 23:35:49.387686 2858 consensus_queue.cc:260] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } } peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } }
I20260810 23:35:49.388989 2859 raft_consensus.cc:348] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } } peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } }
I20260810 23:35:49.389426 2859 raft_consensus.cc:374] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:49.389528 2859 raft_consensus.cc:729] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa970edeeef3483e981d005986bd0951, State: Initialized, Role: FOLLOWER
I20260810 23:35:49.389698 2858 ts_tablet_manager.cc:1428] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260810 23:35:49.390480 2859 consensus_queue.cc:260] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } } peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } }
I20260810 23:35:49.392376 2859 ts_tablet_manager.cc:1428] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260810 23:35:49.392807 2603 heartbeater.cc:503] Master 127.9.48.254:44525 was elected leader, sending a full tablet report...
I20260810 23:35:49.394006 2862 raft_consensus.cc:348] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } } peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } }
I20260810 23:35:49.394511 2862 raft_consensus.cc:374] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:35:49.394619 2862 raft_consensus.cc:729] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229f01207aec449cb3948ad9596890a6, State: Initialized, Role: FOLLOWER
I20260810 23:35:49.395232 2862 consensus_queue.cc:260] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } } peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } }
I20260810 23:35:49.396344 2828 heartbeater.cc:503] Master 127.9.48.254:44525 was elected leader, sending a full tablet report...
I20260810 23:35:49.396986 2862 ts_tablet_manager.cc:1428] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
W20260810 23:35:49.431506 2604 tablet.cc:2369] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:35:49.473847 2866 raft_consensus.cc:482] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:35:49.474308 2866 raft_consensus.cc:504] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } } peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } }
I20260810 23:35:49.476755 2866 leader_election.cc:288] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 837e557a1dde466b948d188ee833077b (127.9.48.193:46583), 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079)
I20260810 23:35:49.484944 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.477088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:39273 (local address 127.9.48.193:46583)
0810 23:35:49.477324 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:49.477332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:49.478005 (+ 673us) server_negotiation.cc:407] Connection header received
0810 23:35:49.478371 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:49.478380 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:49.478492 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:49.478608 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:49.479508 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.480485 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.482768 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.483786 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.483945 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.484004 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:49.484467 (+ 463us) server_negotiation.cc:299] Negotiation successful
0810 23:35:49.484646 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:35:49.485105 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.478337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:35979 (local address 127.9.48.195:44079)
0810 23:35:49.478524 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:49.478530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:49.479189 (+ 659us) server_negotiation.cc:407] Connection header received
0810 23:35:49.479344 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:49.479350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:49.479444 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:49.479538 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:35:49.480076 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.481067 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.482946 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.484034 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.484217 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.484324 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:49.484723 (+ 399us) server_negotiation.cc:299] Negotiation successful
0810 23:35:49.484868 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:35:49.485190 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.478058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:44079 (local address 127.9.48.194:35979)
0810 23:35:49.479045 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:49.479066 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:49.479204 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:49.479567 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:49.479571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:49.479596 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:49.479921 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:49.479929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.481210 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.481214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:49.482748 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:49.482763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.484318 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.484323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:49.484544 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.484600 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 23:35:49.484693 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 23:35:49.484827 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":613,"threads_started":1}
I20260810 23:35:49.485481 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.477087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46583 (local address 127.9.48.194:39273)
0810 23:35:49.477887 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0810 23:35:49.477909 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:35:49.478149 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:35:49.478869 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:35:49.478875 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:35:49.478906 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:35:49.479341 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:49.479355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.480698 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.480704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:49.482597 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:35:49.482612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.484044 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.484051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:35:49.484270 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.484309 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 23:35:49.484419 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 23:35:49.484580 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":436,"threads_started":1}
I20260810 23:35:49.486546 2742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b11fedf70849ac86b0ea8ae9601e57" candidate_uuid: "fa970edeeef3483e981d005986bd0951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229f01207aec449cb3948ad9596890a6" is_pre_election: true
I20260810 23:35:49.486548 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b11fedf70849ac86b0ea8ae9601e57" candidate_uuid: "fa970edeeef3483e981d005986bd0951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "837e557a1dde466b948d188ee833077b" is_pre_election: true
I20260810 23:35:49.487239 2742 raft_consensus.cc:2454] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa970edeeef3483e981d005986bd0951 in term 0.
I20260810 23:35:49.487239 2337 raft_consensus.cc:2454] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa970edeeef3483e981d005986bd0951 in term 0.
I20260810 23:35:49.487998 2439 leader_election.cc:302] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 229f01207aec449cb3948ad9596890a6, fa970edeeef3483e981d005986bd0951; no voters:
I20260810 23:35:49.488384 2866 raft_consensus.cc:2781] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:35:49.488580 2866 raft_consensus.cc:482] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:35:49.488708 2866 raft_consensus.cc:3037] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:49.491039 2866 raft_consensus.cc:504] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } } peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } }
I20260810 23:35:49.491724 2866 leader_election.cc:288] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [CANDIDATE]: Term 1 election: Requested vote from peers 837e557a1dde466b948d188ee833077b (127.9.48.193:46583), 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079)
I20260810 23:35:49.492367 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b11fedf70849ac86b0ea8ae9601e57" candidate_uuid: "fa970edeeef3483e981d005986bd0951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "837e557a1dde466b948d188ee833077b"
I20260810 23:35:49.492437 2742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b11fedf70849ac86b0ea8ae9601e57" candidate_uuid: "fa970edeeef3483e981d005986bd0951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229f01207aec449cb3948ad9596890a6"
I20260810 23:35:49.492642 2337 raft_consensus.cc:3037] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:35:49.492738 2742 raft_consensus.cc:3037] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6 [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:35:49.495013 2831 tablet.cc:2369] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:35:49.495077 2337 raft_consensus.cc:2454] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa970edeeef3483e981d005986bd0951 in term 1.
I20260810 23:35:49.495345 2742 raft_consensus.cc:2454] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa970edeeef3483e981d005986bd0951 in term 1.
I20260810 23:35:49.495623 2439 leader_election.cc:302] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 837e557a1dde466b948d188ee833077b, fa970edeeef3483e981d005986bd0951; no voters:
I20260810 23:35:49.495993 2866 raft_consensus.cc:2781] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:35:49.496583 2866 raft_consensus.cc:686] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [term 1 LEADER]: Becoming Leader. State: Replica: fa970edeeef3483e981d005986bd0951, State: Running, Role: LEADER
I20260810 23:35:49.497129 2866 consensus_queue.cc:237] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } } peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } }
I20260810 23:35:49.502441 2113 catalog_manager.cc:5495] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa970edeeef3483e981d005986bd0951 (127.9.48.194). New cstate: current_term: 1 leader_uuid: "fa970edeeef3483e981d005986bd0951" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } health_report { overall_health: HEALTHY } } }
I20260810 23:35:49.526477 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.520819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.9.48.194:44837)
0810 23:35:49.521060 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:49.521067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:49.521091 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:35:49.521305 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:49.521310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:49.521394 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:49.521528 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:35:49.521987 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.522746 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.524318 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.524804 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.524911 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.525400 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:35:49.525932 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:35:49.526056 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:49.526150 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 23:35:49.526244 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:35:49.535319 2337 raft_consensus.cc:1261] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Refusing update from remote peer fa970edeeef3483e981d005986bd0951: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:35:49.535341 2742 raft_consensus.cc:1261] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6 [term 1 FOLLOWER]: Refusing update from remote peer fa970edeeef3483e981d005986bd0951: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:35:49.536435 2877 consensus_queue.cc:1035] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:35:49.536904 2866 consensus_queue.cc:1035] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:35:49.567526 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:35:49.567873 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:35:49.571666 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Sending DeleteTablet for 3 replicas of tablet 32b11fedf70849ac86b0ea8ae9601e57
I20260810 23:35:49.573093 2713 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b11fedf70849ac86b0ea8ae9601e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:35:49 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:38792
I20260810 23:35:49.573105 2501 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b11fedf70849ac86b0ea8ae9601e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:35:49 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:45002
I20260810 23:35:49.573148 2309 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b11fedf70849ac86b0ea8ae9601e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:35:49 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:60300
I20260810 23:35:49.574826 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 2212
I20260810 23:35:49.589056 2893 tablet_replica.cc:330] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6: stopping tablet replica
I20260810 23:35:49.589696 2893 raft_consensus.cc:2229] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:35:49.590361 2893 raft_consensus.cc:2258] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:35:49.593261 2893 ts_tablet_manager.cc:1905] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:35:49.599668 2894 tablet_replica.cc:330] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b: stopping tablet replica
I20260810 23:35:49.600171 2894 raft_consensus.cc:2229] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:35:49.600771 2894 raft_consensus.cc:2258] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:35:49.601678 2895 tablet_replica.cc:330] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951: stopping tablet replica
I20260810 23:35:49.601644 2893 ts_tablet_manager.cc:1918] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:35:49.601835 2893 log.cc:1196] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/32b11fedf70849ac86b0ea8ae9601e57
I20260810 23:35:49.602334 2895 raft_consensus.cc:2229] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:35:49.602483 2893 ts_tablet_manager.cc:1939] T 32b11fedf70849ac86b0ea8ae9601e57 P 229f01207aec449cb3948ad9596890a6: Deleting consensus metadata
I20260810 23:35:49.603199 2895 raft_consensus.cc:2258] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:35:49.604753 2096 catalog_manager.cc:4841] TS 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079): tablet 32b11fedf70849ac86b0ea8ae9601e57 (table test-table [id=3a1dbe5f0cdb45b08602eb21b8bfc4e9]) successfully deleted
I20260810 23:35:49.605594 2894 ts_tablet_manager.cc:1905] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:35:49.607745 2895 ts_tablet_manager.cc:1905] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 23:35:49 dist-test-slave-04bb krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:35:49.613273 2894 ts_tablet_manager.cc:1918] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:35:49.613476 2894 log.cc:1196] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/32b11fedf70849ac86b0ea8ae9601e57
I20260810 23:35:49.614084 2894 ts_tablet_manager.cc:1939] T 32b11fedf70849ac86b0ea8ae9601e57 P 837e557a1dde466b948d188ee833077b: Deleting consensus metadata
I20260810 23:35:49.616016 2895 ts_tablet_manager.cc:1918] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:35:49.616349 2895 log.cc:1196] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/32b11fedf70849ac86b0ea8ae9601e57
I20260810 23:35:49.616730 2096 catalog_manager.cc:4841] TS 837e557a1dde466b948d188ee833077b (127.9.48.193:46583): tablet 32b11fedf70849ac86b0ea8ae9601e57 (table test-table [id=3a1dbe5f0cdb45b08602eb21b8bfc4e9]) successfully deleted
I20260810 23:35:49.616946 2895 ts_tablet_manager.cc:1939] T 32b11fedf70849ac86b0ea8ae9601e57 P fa970edeeef3483e981d005986bd0951: Deleting consensus metadata
I20260810 23:35:49.618840 2091 catalog_manager.cc:4841] TS fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837): tablet 32b11fedf70849ac86b0ea8ae9601e57 (table test-table [id=3a1dbe5f0cdb45b08602eb21b8bfc4e9]) successfully deleted
I20260810 23:35:49.620803 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.575910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.9.48.193:46583)
0810 23:35:49.576110 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:49.576116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:49.598729 (+ 22613us) server_negotiation.cc:407] Connection header received
0810 23:35:49.598953 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:49.598959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:49.599081 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:49.599231 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:49.601906 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.603171 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.605703 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.606401 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.606533 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.612855 (+ 6322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:49.612876 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:49.612896 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:49.612971 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:49.617273 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:49.618072 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:49.618089 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:49.618092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:49.618221 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:49.619108 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:49.619112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:49.619114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:49.619480 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:49.619808 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:49.620108 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 23:35:49.620333 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":71}
I20260810 23:35:49.621564 2395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:49.713261 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 2212
I20260810 23:35:49.752038 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 2417
Aug 10 23:35:49 dist-test-slave-04bb krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:35:49.771296 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.753050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.9.48.194:44837)
0810 23:35:49.753278 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:49.753284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:49.753308 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:35:49.753532 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:49.753537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:49.753641 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:49.753770 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:49.755561 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.756655 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.758605 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.759170 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.759319 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.764918 (+ 5599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:49.764935 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:49.764952 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:49.764995 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:49.768184 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:49.768984 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:49.768997 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:49.768999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:49.769157 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:49.769581 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:49.769587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:49.769590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:49.769972 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:49.770366 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:49.770676 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 23:35:49.770925 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 23:35:49.772027 2595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:49.887722 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 2417
I20260810 23:35:49.913187 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 2613
Aug 10 23:35:49 dist-test-slave-04bb krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:35:49.929222 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:35:49.913972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38804 (local address 127.9.48.195:44079)
0810 23:35:49.914167 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:35:49.914172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:35:49.914191 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:35:49.914386 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:35:49.914390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:35:49.914468 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:35:49.914558 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:35:49.916115 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.916925 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.918404 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:35:49.918781 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:35:49.918880 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:35:49.923484 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:35:49.923500 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:35:49.923517 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:35:49.923550 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:35:49.926433 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:49.927067 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:49.927075 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:49.927077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:49.927213 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:35:49.927557 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:35:49.927560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:35:49.927562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:35:49.927918 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:35:49.928148 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 23:35:49.928419 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 23:35:49.928930 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:35:49.929944 2818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:50.025691 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 2613
I20260810 23:35:50.053586 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 2064
I20260810 23:35:50.054888 2164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:35:50.171993 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 2064
I20260810 23:36:10.205511 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:44525
--webserver_interface=127.9.48.254
--webserver_port=44461
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:44525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:36:10.567854 2903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:10.568184 2903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:10.568356 2903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:10.575690 2903 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:36:10.575796 2903 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:36:10.575865 2903 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:36:10.575943 2903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:36:10.576041 2903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:10.576170 2903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:10.576247 2903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:36:10.576335 2903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:36:10.586684 2903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:44525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:44525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=44461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:10.588040 2903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:10.589534 2903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:10.616427 2912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:10.616461 2918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:10.616485 2913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:10.617278 2903 server_base.cc:1034] running on GCE node
I20260810 23:36:10.618239 2903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:10.619964 2903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:10.621227 2903 hybrid_clock.cc:648] HybridClock initialized: now 1786404970621145 us; error 50 us; skew 500 ppm
I20260810 23:36:10.621652 2903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:36:10 dist-test-slave-04bb krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404970, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:36:10.625530 2903 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:36:10.627174 2903 webserver.cc:460] Webserver started at http://127.9.48.254:44461/ using document root <none> and password file <none>
I20260810 23:36:10.628033 2903 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:10.628193 2903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:10.634577 2903 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:36:10.638032 2928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:10.639729 2903 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:36:10.639936 2903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "184559be7e544fbeb0b19ac9f7e24f75"
format_stamp: "Formatted at 2026-08-10 23:35:47 on dist-test-slave-04bb"
server_key: "a01df406fde201552a333836f869d2bb"
server_key_iv: "5afb06f08cd3e98ca3ba0f97da81e99b"
server_key_version: "encryptionkey@0"
I20260810 23:36:10.641243 2903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:10.685897 2903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:10.691421 2903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:10.691812 2903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:10.713757 2903 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:44525
I20260810 23:36:10.713800 3004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:44525 every 8 connection(s)
I20260810 23:36:10.715547 2903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:36:10.720983 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 2903
I20260810 23:36:10.722098 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:46583
--local_ip_for_outbound_sockets=127.9.48.193
--tserver_master_addrs=127.9.48.254:44525
--webserver_port=35867
--webserver_interface=127.9.48.193
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:36:10.724138 3005 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:36:10.729771 3005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Bootstrap starting.
I20260810 23:36:10.752635 3005 log.cc:824] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Log is configured to *not* fsync() on all Append() calls
I20260810 23:36:10.765550 3005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:36:10.766162 3005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Bootstrap complete.
I20260810 23:36:10.775820 3005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } }
I20260810 23:36:10.776801 3005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184559be7e544fbeb0b19ac9f7e24f75, State: Initialized, Role: FOLLOWER
I20260810 23:36:10.777535 3005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } }
I20260810 23:36:10.777809 3005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:36:10.777948 3005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:36:10.778139 3005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:36:10.782274 3005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } }
I20260810 23:36:10.782878 3005 leader_election.cc:302] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 184559be7e544fbeb0b19ac9f7e24f75; no voters:
I20260810 23:36:10.783571 3005 leader_election.cc:288] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:36:10.783715 3011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:36:10.785202 3011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 2 LEADER]: Becoming Leader. State: Replica: 184559be7e544fbeb0b19ac9f7e24f75, State: Running, Role: LEADER
I20260810 23:36:10.785883 3011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } }
I20260810 23:36:10.786154 3005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:36:10.788757 3012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "184559be7e544fbeb0b19ac9f7e24f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } } }
I20260810 23:36:10.789153 3012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [sys.catalog]: This master's current role is: LEADER
I20260810 23:36:10.789650 3013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 184559be7e544fbeb0b19ac9f7e24f75. Latest consensus state: current_term: 2 leader_uuid: "184559be7e544fbeb0b19ac9f7e24f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } } }
I20260810 23:36:10.789968 3013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [sys.catalog]: This master's current role is: LEADER
I20260810 23:36:10.794878 3017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:36:10.805096 3017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:36:10.807005 3017 catalog_manager.cc:1259] Loaded cluster ID: 1ffbf61718644dabbf84140370f53026
I20260810 23:36:10.807217 3017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:36:10.812976 3017 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:36:10.815115 3017 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Loaded TSK: 0
I20260810 23:36:10.817095 3032 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:36:10.841239 3017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Generated new TSK 1
I20260810 23:36:10.841663 3017 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:36:11.121636 3009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:11.122040 3009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:11.122258 3009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:11.134913 3009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:11.135078 3009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:11.135272 3009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:36:11.146461 3009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:46583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=35867
--tserver_master_addrs=127.9.48.254:44525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:11.147982 3009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:11.149528 3009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:11.164302 3046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:11.164461 3045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:11.164491 3048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:11.165269 3009 server_base.cc:1034] running on GCE node
I20260810 23:36:11.166227 3009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:11.167565 3009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:11.168816 3009 hybrid_clock.cc:648] HybridClock initialized: now 1786404971168732 us; error 70 us; skew 500 ppm
I20260810 23:36:11.169210 3009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:36:11 dist-test-slave-04bb krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404971, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:36:11.173388 3009 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:36:11.175137 3009 webserver.cc:460] Webserver started at http://127.9.48.193:35867/ using document root <none> and password file <none>
I20260810 23:36:11.175918 3009 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:11.176090 3009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:11.182050 3009 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:36:11.185196 3061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:11.186769 3009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 23:36:11.187001 3009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "837e557a1dde466b948d188ee833077b"
format_stamp: "Formatted at 2026-08-10 23:35:48 on dist-test-slave-04bb"
server_key: "f4037bb04a904f875770deaa19634779"
server_key_iv: "96d6c1e128935fec4fcb3519ba6494e2"
server_key_version: "encryptionkey@0"
I20260810 23:36:11.188159 3009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:11.211349 3009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:11.215996 3009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:11.216298 3009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:11.217365 3009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:36:11.218771 3009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:36:11.218896 3009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:11.219040 3009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:36:11.219132 3009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:11.253791 3009 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:46583
I20260810 23:36:11.253882 3228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:46583 every 8 connection(s)
I20260810 23:36:11.255379 3009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:36:11.263355 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 3009
I20260810 23:36:11.264216 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:44837
--local_ip_for_outbound_sockets=127.9.48.194
--tserver_master_addrs=127.9.48.254:44525
--webserver_port=37693
--webserver_interface=127.9.48.194
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:36:11 dist-test-slave-04bb krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404971, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:36:11.282399 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:11.258275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:59443 (local address 127.9.48.254:44525)
0810 23:36:11.258887 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:11.258899 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:11.259444 (+ 545us) server_negotiation.cc:407] Connection header received
0810 23:36:11.261667 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:11.261679 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:11.262030 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:11.262336 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:11.263959 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:11.266150 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:11.268098 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:11.269220 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:11.269434 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:11.273148 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:11.273161 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:11.273184 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:11.273245 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:11.277513 (+ 4268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:11.278527 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:11.278534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:11.278540 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:11.278697 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:11.279255 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:11.279259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:11.279261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:11.279733 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:11.279895 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:11.280948 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0810 23:36:11.281278 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":265,"threads_started":1}
I20260810 23:36:11.282776 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:11.258407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:59443)
0810 23:36:11.259253 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:11.259309 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:11.261178 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:11.262499 (+ 1321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:11.262507 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:11.263032 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:36:11.263654 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:11.263676 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:11.266359 (+ 2683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:11.266363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:11.267928 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:11.267943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:11.269493 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:11.269499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:11.269717 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:11.270482 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:36:11.270512 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:36:11.272847 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:36:11.277671 (+ 4824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:11.277683 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:11.277702 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:11.278386 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:11.278832 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:11.278840 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:11.278846 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:11.279143 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:11.279905 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:36:11.279917 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:36:11.280397 (+ 480us) client_negotiation.cc:769] Sending connection context
0810 23:36:11.280737 (+ 340us) client_negotiation.cc:240] Negotiation successful
0810 23:36:11.281366 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":375,"threads_started":1}
I20260810 23:36:11.286185 3229 heartbeater.cc:347] Connected to a master server at 127.9.48.254:44525
I20260810 23:36:11.286520 3229 heartbeater.cc:464] Registering TS with master...
I20260810 23:36:11.287369 3229 heartbeater.cc:511] Master 127.9.48.254:44525 requested a full tablet report, sending...
I20260810 23:36:11.290126 2955 ts_manager.cc:194] Registered new tserver with Master: 837e557a1dde466b948d188ee833077b (127.9.48.193:46583)
I20260810 23:36:11.292414 2955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:59443
W20260810 23:36:11.625638 3235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:11.625986 3235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:11.626174 3235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:11.633673 3235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:11.633857 3235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:11.634040 3235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:36:11.645931 3235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:44837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=37693
--tserver_master_addrs=127.9.48.254:44525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:11.647416 3235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:11.648967 3235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:11.662693 3241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:11.662817 3242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:11.663035 3248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:11.664175 3235 server_base.cc:1034] running on GCE node
I20260810 23:36:11.664899 3235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:11.665925 3235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:11.667143 3235 hybrid_clock.cc:648] HybridClock initialized: now 1786404971667096 us; error 36 us; skew 500 ppm
I20260810 23:36:11.667461 3235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:36:11 dist-test-slave-04bb krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404971, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:36:11.670924 3235 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:36:11.672693 3235 webserver.cc:460] Webserver started at http://127.9.48.194:37693/ using document root <none> and password file <none>
I20260810 23:36:11.673451 3235 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:11.673625 3235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:11.679350 3235 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:36:11.682637 3260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:11.684245 3235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:36:11.684446 3235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "fa970edeeef3483e981d005986bd0951"
format_stamp: "Formatted at 2026-08-10 23:35:48 on dist-test-slave-04bb"
server_key: "ced851f212b35e5de7496a884cfcf256"
server_key_iv: "75693b5aa77aabb8ec46aa4bc54e162e"
server_key_version: "encryptionkey@0"
I20260810 23:36:11.685518 3235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:11.707659 3235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:11.712291 3235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:11.712596 3235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:11.713863 3235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:36:11.715626 3235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:36:11.715739 3235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:11.715842 3235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:36:11.715919 3235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:11.754644 3235 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:44837
I20260810 23:36:11.754727 3421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:44837 every 8 connection(s)
I20260810 23:36:11.756158 3235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:36:11.762732 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 3235
I20260810 23:36:11.763415 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:44079
--local_ip_for_outbound_sockets=127.9.48.195
--tserver_master_addrs=127.9.48.254:44525
--webserver_port=33069
--webserver_interface=127.9.48.195
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:36:11 dist-test-slave-04bb krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404971, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:36:11.777019 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:11.759081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:55949 (local address 127.9.48.254:44525)
0810 23:36:11.759323 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:11.759328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:11.760590 (+ 1262us) server_negotiation.cc:407] Connection header received
0810 23:36:11.762541 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:11.762547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:11.762667 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:11.762800 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:11.764469 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:11.765272 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:11.766845 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:11.767293 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:11.767435 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:11.771045 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:11.771058 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:11.771062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:11.771094 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:11.773792 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:11.774537 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:11.774545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:11.774551 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:11.774690 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:11.775127 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:11.775132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:11.775136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:11.775421 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:11.775583 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:11.776342 (+ 759us) server_negotiation.cc:299] Negotiation successful
0810 23:36:11.776630 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 23:36:11.778059 3426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:11.759589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:55949)
0810 23:36:11.760381 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:11.760431 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:11.762242 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:11.763060 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:11.763068 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:11.763650 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:36:11.764273 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:11.764292 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:11.765455 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:11.765460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:11.766688 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:11.766701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:11.767548 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:11.767556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:11.767793 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:11.768574 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:36:11.768615 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:36:11.770802 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:36:11.773946 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:11.773956 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:11.773972 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:11.774409 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:11.774809 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:11.774814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:11.774817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:11.775009 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:11.775558 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:36:11.775566 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:36:11.775973 (+ 407us) client_negotiation.cc:769] Sending connection context
0810 23:36:11.776258 (+ 285us) client_negotiation.cc:240] Negotiation successful
0810 23:36:11.776864 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":331,"threads_started":1}
I20260810 23:36:11.779760 3422 heartbeater.cc:347] Connected to a master server at 127.9.48.254:44525
I20260810 23:36:11.780160 3422 heartbeater.cc:464] Registering TS with master...
I20260810 23:36:11.781141 3422 heartbeater.cc:511] Master 127.9.48.254:44525 requested a full tablet report, sending...
I20260810 23:36:11.782958 2955 ts_manager.cc:194] Registered new tserver with Master: fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837)
I20260810 23:36:11.784354 2955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:55949
W20260810 23:36:12.134089 3427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:12.134388 3427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:12.134485 3427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:12.142133 3427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:12.142256 3427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:12.142377 3427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:36:12.152833 3427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:44079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=33069
--tserver_master_addrs=127.9.48.254:44525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:12.154212 3427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:12.155598 3427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:12.169140 3433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:12.169183 3438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:12.169301 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:12.170293 3427 server_base.cc:1034] running on GCE node
I20260810 23:36:12.170954 3427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:12.171909 3427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:12.173141 3427 hybrid_clock.cc:648] HybridClock initialized: now 1786404972173073 us; error 67 us; skew 500 ppm
I20260810 23:36:12.173473 3427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:36:12 dist-test-slave-04bb krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404972, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:36:12.177002 3427 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:36:12.178534 3427 webserver.cc:460] Webserver started at http://127.9.48.195:33069/ using document root <none> and password file <none>
I20260810 23:36:12.179283 3427 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:12.179411 3427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:12.185781 3427 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 23:36:12.188989 3450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:12.190783 3427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:36:12.190940 3427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "229f01207aec449cb3948ad9596890a6"
format_stamp: "Formatted at 2026-08-10 23:35:49 on dist-test-slave-04bb"
server_key: "ace0af1a573b449577c7561794a865c6"
server_key_iv: "e76d9bf63d249ce9f6aeb6cc88ce945d"
server_key_version: "encryptionkey@0"
I20260810 23:36:12.192168 3427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:12.236071 3427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:12.241200 3427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:12.241492 3427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:12.242533 3427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:36:12.243979 3427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:36:12.244092 3427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:12.244180 3427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:36:12.244254 3427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:12.282757 3427 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:44079
I20260810 23:36:12.282855 3613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:44079 every 8 connection(s)
I20260810 23:36:12.284375 3427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:36:12.292479 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 3427
I20260810 23:36:12.296092 3229 heartbeater.cc:503] Master 127.9.48.254:44525 was elected leader, sending a full tablet report...
Aug 10 23:36:12 dist-test-slave-04bb krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404972, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:36:12.308772 3619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.287172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:33663 (local address 127.9.48.254:44525)
0810 23:36:12.287675 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:12.287680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:12.288607 (+ 927us) server_negotiation.cc:407] Connection header received
0810 23:36:12.290555 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:12.290563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:12.290661 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:12.290786 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:12.292847 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.293726 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.296034 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.296635 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.296760 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.301096 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:12.301105 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:12.301110 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:12.301159 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:12.305013 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:12.305995 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:12.306000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:12.306002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:12.306143 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:12.306688 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:12.306694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:12.306698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:12.307045 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:12.307164 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:12.308137 (+ 973us) server_negotiation.cc:299] Negotiation successful
0810 23:36:12.308355 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"spinlock_wait_cycles":13568,"thread_start_us":277,"threads_started":1}
I20260810 23:36:12.310127 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.287659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:33663)
0810 23:36:12.288401 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:12.288453 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:12.290257 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:12.291052 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:12.291062 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:12.291666 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:36:12.292559 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:12.292592 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.294017 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.294025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:12.295886 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:12.295902 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.297129 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.297138 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:12.297437 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.298288 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:36:12.298324 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:36:12.300838 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:36:12.305153 (+ 4315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:12.305163 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:12.305182 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:12.305856 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:12.306261 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:12.306267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:12.306271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:12.306555 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:12.307191 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:36:12.307202 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:36:12.307694 (+ 492us) client_negotiation.cc:769] Sending connection context
0810 23:36:12.308058 (+ 364us) client_negotiation.cc:240] Negotiation successful
0810 23:36:12.308702 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":283,"threads_started":1}
I20260810 23:36:12.311611 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.293591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53426 (local address 127.9.48.254:44525)
0810 23:36:12.294852 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:12.294857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:12.294875 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:36:12.294922 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:12.294927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:12.294996 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:12.295112 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:12.297438 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.298564 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.300034 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.300716 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.300859 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.304919 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:12.304924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:12.304927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:12.304988 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:12.308468 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:12.309350 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:12.309356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:12.309359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:12.309488 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:12.309889 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:12.309894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:12.309897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:12.310286 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:12.310657 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:12.311045 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 23:36:12.311250 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1132,"spinlock_wait_cycles":4864,"thread_start_us":1003,"threads_started":1}
I20260810 23:36:12.312072 3616 heartbeater.cc:347] Connected to a master server at 127.9.48.254:44525
I20260810 23:36:12.312455 3616 heartbeater.cc:464] Registering TS with master...
I20260810 23:36:12.313365 3616 heartbeater.cc:511] Master 127.9.48.254:44525 requested a full tablet report, sending...
I20260810 23:36:12.314793 2955 ts_manager.cc:194] Registered new tserver with Master: 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079)
I20260810 23:36:12.315550 2955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:33663
I20260810 23:36:12.327126 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:36:12.333772 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.328196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.9.48.254:44525)
0810 23:36:12.328413 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:12.328418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:12.328838 (+ 420us) server_negotiation.cc:407] Connection header received
0810 23:36:12.329035 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:12.329038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:12.329205 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:12.329363 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:36:12.329763 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.330456 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.331870 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.332324 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.332421 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.332970 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:36:12.333257 (+ 287us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 23:36:12.333567 (+ 310us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53434: token expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 23:36:12.333882 3622 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53434: token expired
W20260810 23:36:12.334525 2835 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.48.254:44525)
I20260810 23:36:12.347898 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.335062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.9.48.254:44525)
0810 23:36:12.335214 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:12.335217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:12.335285 (+ 68us) server_negotiation.cc:407] Connection header received
0810 23:36:12.335405 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:12.335408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:12.335467 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:12.335542 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:12.336988 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.337726 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.339022 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.339424 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.339517 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.342438 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:12.342443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:12.342446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:12.342480 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:12.345380 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:12.346100 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:12.346107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:12.346112 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:12.346257 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:12.346611 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:12.346614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:12.346616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:12.346938 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:12.347129 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:12.347459 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 23:36:12.347637 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:36:12.351564 2955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:36:12.359871 2955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:36:12.383562 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.372707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:44079 (local address 127.0.0.1:36088)
0810 23:36:12.373632 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:12.373679 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:12.373839 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:12.375817 (+ 1978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:12.375824 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:12.375856 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:36:12.376581 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:12.376606 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.378935 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.378943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:12.380986 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:12.380999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.382479 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.382486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:12.382713 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.382780 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 23:36:12.382931 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 23:36:12.383119 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":544,"threads_started":1}
I20260810 23:36:12.384797 3637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.373093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.9.48.195:44079)
0810 23:36:12.375097 (+ 2004us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:12.375110 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:12.375149 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:36:12.375275 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:12.375284 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:12.375552 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:12.376723 (+ 1171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:36:12.377064 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.378804 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.381154 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.382190 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.382355 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.382669 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:12.382995 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 23:36:12.384456 (+ 1461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1814,"thread_start_us":1654,"threads_started":1}
I20260810 23:36:12.386205 3638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.374260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55428 (local address 127.9.48.194:44837)
0810 23:36:12.375539 (+ 1279us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:12.375552 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:12.376857 (+ 1305us) server_negotiation.cc:407] Connection header received
0810 23:36:12.377212 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:12.377220 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:12.377466 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:12.377644 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:36:12.378700 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.380305 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.383493 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.385024 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.385163 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.385367 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:12.385676 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 23:36:12.385852 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1052,"thread_start_us":1293,"threads_started":1}
I20260810 23:36:12.386271 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.372979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:44837 (local address 127.0.0.1:55428)
0810 23:36:12.376719 (+ 3740us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:12.376745 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:12.376984 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:12.377946 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:12.377951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:12.377997 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:36:12.378527 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:12.378539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.380461 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.380466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:12.383305 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:12.383319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.385277 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.385285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:12.385472 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.385508 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:36:12.385612 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 23:36:12.385750 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3600,"mutex_wait_us":172,"spinlock_wait_cycles":1920,"thread_start_us":3385,"threads_started":1}
I20260810 23:36:12.387100 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.375365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.9.48.193:46583)
0810 23:36:12.376116 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:12.376128 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:12.377495 (+ 1367us) server_negotiation.cc:407] Connection header received
0810 23:36:12.377595 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:12.377600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:12.377758 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:12.377944 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:36:12.378552 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.380281 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.383493 (+ 3212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.385024 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.385163 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.385391 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:12.386568 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0810 23:36:12.386763 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":407,"threads_started":1}
I20260810 23:36:12.387429 3640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.374744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46583 (local address 127.0.0.1:40640)
0810 23:36:12.377252 (+ 2508us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:12.377270 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:12.377384 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:12.377936 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:12.377941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:12.377966 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:36:12.378380 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:12.378390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.380462 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.380466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:12.383304 (+ 2838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:12.383319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.385537 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.385542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:12.385743 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.385777 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:36:12.385900 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 23:36:12.386042 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2374,"mutex_wait_us":131,"thread_start_us":2234,"threads_started":1}
I20260810 23:36:12.389245 3517 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac246863fef4d2a8da2a114f0281d68 (DEFAULT_TABLE table=test-table [id=be20c97ed83a494996515133afb5a309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:36:12.390246 3328 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac246863fef4d2a8da2a114f0281d68 (DEFAULT_TABLE table=test-table [id=be20c97ed83a494996515133afb5a309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:36:12.391181 3517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac246863fef4d2a8da2a114f0281d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:36:12.390821 3134 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac246863fef4d2a8da2a114f0281d68 (DEFAULT_TABLE table=test-table [id=be20c97ed83a494996515133afb5a309]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:36:12.391680 3328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac246863fef4d2a8da2a114f0281d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:36:12.392233 3134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac246863fef4d2a8da2a114f0281d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:36:12.404919 3646 tablet_bootstrap.cc:492] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b: Bootstrap starting.
I20260810 23:36:12.407434 3647 tablet_bootstrap.cc:492] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951: Bootstrap starting.
I20260810 23:36:12.408324 3648 tablet_bootstrap.cc:492] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6: Bootstrap starting.
I20260810 23:36:12.409165 3646 tablet_bootstrap.cc:654] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b: Neither blocks nor log segments found. Creating new log.
I20260810 23:36:12.410630 3646 log.cc:824] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:36:12.412153 3647 tablet_bootstrap.cc:654] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951: Neither blocks nor log segments found. Creating new log.
I20260810 23:36:12.413520 3647 log.cc:824] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951: Log is configured to *not* fsync() on all Append() calls
I20260810 23:36:12.414474 3646 tablet_bootstrap.cc:492] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b: No bootstrap required, opened a new log
I20260810 23:36:12.414780 3646 ts_tablet_manager.cc:1397] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260810 23:36:12.414944 3648 tablet_bootstrap.cc:654] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6: Neither blocks nor log segments found. Creating new log.
I20260810 23:36:12.416209 3648 log.cc:824] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:36:12.416846 3647 tablet_bootstrap.cc:492] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951: No bootstrap required, opened a new log
I20260810 23:36:12.417164 3647 ts_tablet_manager.cc:1397] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260810 23:36:12.419557 3648 tablet_bootstrap.cc:492] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6: No bootstrap required, opened a new log
I20260810 23:36:12.419838 3648 ts_tablet_manager.cc:1397] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260810 23:36:12.422948 3646 raft_consensus.cc:348] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:12.423346 3646 raft_consensus.cc:374] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:36:12.423487 3646 raft_consensus.cc:729] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 837e557a1dde466b948d188ee833077b, State: Initialized, Role: FOLLOWER
I20260810 23:36:12.424124 3646 consensus_queue.cc:260] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:12.426021 3646 ts_tablet_manager.cc:1428] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260810 23:36:12.428069 3647 raft_consensus.cc:348] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:12.428586 3647 raft_consensus.cc:374] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:36:12.428711 3647 raft_consensus.cc:729] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa970edeeef3483e981d005986bd0951, State: Initialized, Role: FOLLOWER
I20260810 23:36:12.429518 3647 consensus_queue.cc:260] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:12.430909 3648 raft_consensus.cc:348] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:12.431501 3648 raft_consensus.cc:374] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:36:12.431644 3648 raft_consensus.cc:729] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229f01207aec449cb3948ad9596890a6, State: Initialized, Role: FOLLOWER
I20260810 23:36:12.431820 3422 heartbeater.cc:503] Master 127.9.48.254:44525 was elected leader, sending a full tablet report...
I20260810 23:36:12.432638 3647 ts_tablet_manager.cc:1428] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951: Time spent starting tablet: real 0.015s user 0.005s sys 0.010s
I20260810 23:36:12.432479 3648 consensus_queue.cc:260] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:12.434412 3648 ts_tablet_manager.cc:1428] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260810 23:36:12.434558 3616 heartbeater.cc:503] Master 127.9.48.254:44525 was elected leader, sending a full tablet report...
W20260810 23:36:12.507949 3231 tablet.cc:2369] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:36:12.508405 3423 tablet.cc:2369] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:36:12.536142 3618 tablet.cc:2369] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:36:12.622226 3658 raft_consensus.cc:482] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:12.622578 3658 raft_consensus.cc:504] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:12.624434 3658 leader_election.cc:288] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079), 837e557a1dde466b948d188ee833077b (127.9.48.193:46583)
I20260810 23:36:12.633817 3637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.624969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:37617 (local address 127.9.48.195:44079)
0810 23:36:12.625173 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:12.625178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:12.625767 (+ 589us) server_negotiation.cc:407] Connection header received
0810 23:36:12.625990 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:12.625995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:12.626078 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:12.626186 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:36:12.627034 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.628125 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.630941 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.632466 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.632728 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.632822 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:12.633382 (+ 560us) server_negotiation.cc:299] Negotiation successful
0810 23:36:12.633565 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:36:12.633805 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.626068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:46663 (local address 127.9.48.193:46583)
0810 23:36:12.626292 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:12.626297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:12.626531 (+ 234us) server_negotiation.cc:407] Connection header received
0810 23:36:12.626709 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:12.626713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:12.626770 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:12.626843 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:36:12.627417 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.628446 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.630941 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.632466 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.632722 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.632822 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:12.633382 (+ 560us) server_negotiation.cc:299] Negotiation successful
0810 23:36:12.633530 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:36:12.634052 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.624806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:44079 (local address 127.9.48.194:37617)
0810 23:36:12.625614 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:12.625638 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:12.625793 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:12.626351 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:12.626356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:12.626382 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:36:12.626872 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:12.626885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.628266 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.628270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:12.630687 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:12.630704 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.632798 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.632805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:12.633067 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.633198 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 23:36:12.633319 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 23:36:12.633517 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":150,"spinlock_wait_cycles":38656,"thread_start_us":422,"threads_started":1}
I20260810 23:36:12.634075 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.625750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46583 (local address 127.9.48.194:46663)
0810 23:36:12.626417 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:12.626439 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:12.626586 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:12.626890 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:12.626894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:12.626921 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:36:12.627253 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:12.627261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.628596 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.628601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:12.630743 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:12.630756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.632807 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.632813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:12.633067 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.633198 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 23:36:12.633301 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:36:12.633519 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":136,"thread_start_us":366,"threads_started":1}
I20260810 23:36:12.635303 3547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac246863fef4d2a8da2a114f0281d68" candidate_uuid: "fa970edeeef3483e981d005986bd0951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229f01207aec449cb3948ad9596890a6" is_pre_election: true
I20260810 23:36:12.635311 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac246863fef4d2a8da2a114f0281d68" candidate_uuid: "fa970edeeef3483e981d005986bd0951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "837e557a1dde466b948d188ee833077b" is_pre_election: true
I20260810 23:36:12.635845 3547 raft_consensus.cc:2454] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa970edeeef3483e981d005986bd0951 in term 0.
I20260810 23:36:12.635845 3165 raft_consensus.cc:2454] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa970edeeef3483e981d005986bd0951 in term 0.
I20260810 23:36:12.636605 3266 leader_election.cc:302] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 229f01207aec449cb3948ad9596890a6, fa970edeeef3483e981d005986bd0951; no voters:
I20260810 23:36:12.637022 3658 raft_consensus.cc:2781] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:36:12.637180 3658 raft_consensus.cc:482] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:36:12.637298 3658 raft_consensus.cc:3037] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:36:12.639550 3658 raft_consensus.cc:504] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:12.640210 3658 leader_election.cc:288] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [CANDIDATE]: Term 1 election: Requested vote from peers 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079), 837e557a1dde466b948d188ee833077b (127.9.48.193:46583)
I20260810 23:36:12.640815 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac246863fef4d2a8da2a114f0281d68" candidate_uuid: "fa970edeeef3483e981d005986bd0951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "837e557a1dde466b948d188ee833077b"
I20260810 23:36:12.640820 3547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac246863fef4d2a8da2a114f0281d68" candidate_uuid: "fa970edeeef3483e981d005986bd0951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "229f01207aec449cb3948ad9596890a6"
I20260810 23:36:12.641058 3547 raft_consensus.cc:3037] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:36:12.641052 3165 raft_consensus.cc:3037] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:36:12.644435 3547 raft_consensus.cc:2454] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa970edeeef3483e981d005986bd0951 in term 1.
I20260810 23:36:12.644435 3165 raft_consensus.cc:2454] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa970edeeef3483e981d005986bd0951 in term 1.
I20260810 23:36:12.644913 3266 leader_election.cc:302] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 229f01207aec449cb3948ad9596890a6, fa970edeeef3483e981d005986bd0951; no voters:
I20260810 23:36:12.645309 3658 raft_consensus.cc:2781] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:36:12.645910 3658 raft_consensus.cc:686] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [term 1 LEADER]: Becoming Leader. State: Replica: fa970edeeef3483e981d005986bd0951, State: Running, Role: LEADER
I20260810 23:36:12.646597 3658 consensus_queue.cc:237] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:12.651815 2953 catalog_manager.cc:5495] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa970edeeef3483e981d005986bd0951 (127.9.48.194). New cstate: current_term: 1 leader_uuid: "fa970edeeef3483e981d005986bd0951" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } health_report { overall_health: UNKNOWN } } }
I20260810 23:36:12.687831 3638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.682096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.9.48.194:44837)
0810 23:36:12.682323 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:12.682329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:12.682349 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:36:12.682602 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:12.682607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:12.682690 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:12.682869 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:36:12.683248 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.683981 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.685474 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.685898 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.686044 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.686685 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:36:12.687202 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:36:12.687398 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:12.687449 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 23:36:12.687531 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:36:12.696893 3165 raft_consensus.cc:1261] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Refusing update from remote peer fa970edeeef3483e981d005986bd0951: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:36:12.696892 3547 raft_consensus.cc:1261] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6 [term 1 FOLLOWER]: Refusing update from remote peer fa970edeeef3483e981d005986bd0951: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:36:12.697961 3658 consensus_queue.cc:1035] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:36:12.698289 3665 consensus_queue.cc:1035] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:36:12.714078 3672 mvcc.cc:205] Tried to move back new op lower bound from 7317114768155074560 to 7317114767965798400. Current Snapshot: MvccSnapshot[applied={T|T < 7317114768155074560}]
I20260810 23:36:12.727595 2955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:36:12.727906 2955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:36:12.731487 2955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Sending DeleteTablet for 3 replicas of tablet 1ac246863fef4d2a8da2a114f0281d68
I20260810 23:36:12.733109 3328 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac246863fef4d2a8da2a114f0281d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:36:12 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:55428
I20260810 23:36:12.733160 3517 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac246863fef4d2a8da2a114f0281d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:36:12 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:36088
I20260810 23:36:12.733160 3134 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac246863fef4d2a8da2a114f0281d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:36:12 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:40640
I20260810 23:36:12.734655 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 3009
I20260810 23:36:12.748821 3683 tablet_replica.cc:330] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6: stopping tablet replica
I20260810 23:36:12.749701 3687 tablet_replica.cc:330] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951: stopping tablet replica
I20260810 23:36:12.750327 3687 raft_consensus.cc:2229] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:36:12.750927 3683 raft_consensus.cc:2229] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:36:12.751742 3687 raft_consensus.cc:2258] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:36:12.752369 3683 raft_consensus.cc:2258] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:36:12.755312 3683 ts_tablet_manager.cc:1905] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:36:12.756764 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.736662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40646 (local address 127.9.48.193:46583)
0810 23:36:12.736853 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:12.736862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:12.736884 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:36:12.736944 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:12.736950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:12.737046 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:12.737159 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:12.739259 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.740453 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.742470 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.743117 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.743254 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.747503 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:12.747525 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:12.747544 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:12.747596 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:12.752045 (+ 4449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:12.753181 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:12.753195 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:12.753202 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:12.753450 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:12.754024 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:12.754032 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:12.754038 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:12.754616 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:12.754880 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:12.756108 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0810 23:36:12.756347 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:36:12.757370 3687 ts_tablet_manager.cc:1905] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:36:12.757761 3223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:36:12.764959 3687 ts_tablet_manager.cc:1918] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:36:12.772207 3683 ts_tablet_manager.cc:1918] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:36:12.772477 3687 log.cc:1196] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/1ac246863fef4d2a8da2a114f0281d68
I20260810 23:36:12.772787 3683 log.cc:1196] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/1ac246863fef4d2a8da2a114f0281d68
I20260810 23:36:12.773177 3687 ts_tablet_manager.cc:1939] T 1ac246863fef4d2a8da2a114f0281d68 P fa970edeeef3483e981d005986bd0951: Deleting consensus metadata
I20260810 23:36:12.773351 3683 ts_tablet_manager.cc:1939] T 1ac246863fef4d2a8da2a114f0281d68 P 229f01207aec449cb3948ad9596890a6: Deleting consensus metadata
I20260810 23:36:12.775286 2930 catalog_manager.cc:4841] TS fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837): tablet 1ac246863fef4d2a8da2a114f0281d68 (table test-table [id=be20c97ed83a494996515133afb5a309]) successfully deleted
I20260810 23:36:12.775650 2934 catalog_manager.cc:4841] TS 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079): tablet 1ac246863fef4d2a8da2a114f0281d68 (table test-table [id=be20c97ed83a494996515133afb5a309]) successfully deleted
I20260810 23:36:12.870576 3682 tablet_replica.cc:330] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b: stopping tablet replica
I20260810 23:36:12.870754 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 3009
W20260810 23:36:12.901329 2934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.48.193:46583 (error 108)
I20260810 23:36:12.903509 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 3235
I20260810 23:36:12.905238 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.903891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46583 (local address 127.0.0.1:40648)
0810 23:36:12.904957 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:12.905065 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":931,"thread_start_us":818,"threads_started":1}
W20260810 23:36:12.905556 2934 catalog_manager.cc:4584] TS 837e557a1dde466b948d188ee833077b (127.9.48.193:46583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ac246863fef4d2a8da2a114f0281d68: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46583: connect: Connection refused (error 111)
I20260810 23:36:12.921218 3638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.904376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.9.48.194:44837)
0810 23:36:12.904620 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:12.904629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:12.905084 (+ 455us) server_negotiation.cc:407] Connection header received
0810 23:36:12.905159 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:12.905165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:12.905266 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:12.905369 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:12.907447 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.908309 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.910413 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:12.910868 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:12.910973 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:12.914691 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:12.914708 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:12.914723 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:12.914766 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:12.918232 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:12.919085 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:12.919093 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:12.919095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:12.919233 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:12.919656 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:12.919659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:12.919661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:12.920037 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:12.920292 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:12.920694 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 23:36:12.920874 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:36:12.922048 3416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:36:12.959555 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.959019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46583 (local address 127.0.0.1:40664)
0810 23:36:12.959282 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:12.959394 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260810 23:36:12.960762 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:12.960342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46583 (local address 127.0.0.1:40680)
0810 23:36:12.960497 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:12.960625 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 23:36:13.021306 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:13.020836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46583 (local address 127.0.0.1:40694)
0810 23:36:13.021061 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:13.021151 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 23:36:13.022405 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:13.022025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46583 (local address 127.0.0.1:40706)
0810 23:36:13.022189 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:13.022263 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 23:36:13.040060 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 3235
I20260810 23:36:13.069665 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 3427
I20260810 23:36:13.084124 3637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:13.070472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.9.48.195:44079)
0810 23:36:13.070666 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:13.070675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:13.070781 (+ 106us) server_negotiation.cc:407] Connection header received
0810 23:36:13.071005 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:13.071015 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:13.071087 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:13.071194 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:13.072602 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:13.073332 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:13.074770 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:13.075165 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:13.075316 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:13.078155 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:13.078171 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:13.078188 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:13.078231 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:13.081099 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:13.081881 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:13.081889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:13.081891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:13.082041 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:13.082488 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:13.082498 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:13.082501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:13.082872 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:13.083187 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:13.083507 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 23:36:13.083742 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:36:13.084941 3608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:36:13.103120 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:13.102512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46583 (local address 127.0.0.1:40716)
0810 23:36:13.102805 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:13.102923 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260810 23:36:13.104359 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:13.103862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:46583 (local address 127.0.0.1:40724)
0810 23:36:13.104094 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:13.104186 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:46583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260810 23:36:13.177681 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 3427
I20260810 23:36:13.202383 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 2903
I20260810 23:36:13.203751 2998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:36:13.320703 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 2903
I20260810 23:36:33.355427 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:44525
--webserver_interface=127.9.48.254
--webserver_port=44461
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:44525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:36:33.715490 3696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:33.715781 3696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:33.715991 3696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:33.724682 3696 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:36:33.724784 3696 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:36:33.724867 3696 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:36:33.724951 3696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:36:33.725052 3696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:33.725165 3696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:33.725231 3696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:36:33.725327 3696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:36:33.736205 3696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:44525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:44525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=44461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:33.737663 3696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:33.739183 3696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:33.752224 3703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:33.752279 3705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:33.752226 3702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:33.753198 3696 server_base.cc:1034] running on GCE node
I20260810 23:36:33.754179 3696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:33.755702 3696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:33.756947 3696 hybrid_clock.cc:648] HybridClock initialized: now 1786404993756865 us; error 43 us; skew 500 ppm
I20260810 23:36:33.757393 3696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:36:33 dist-test-slave-04bb krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404993, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:36:33.761157 3696 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:36:33.762812 3696 webserver.cc:460] Webserver started at http://127.9.48.254:44461/ using document root <none> and password file <none>
I20260810 23:36:33.763602 3696 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:33.763761 3696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:33.769960 3696 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:36:33.772962 3714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:33.774336 3696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:36:33.774531 3696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "184559be7e544fbeb0b19ac9f7e24f75"
format_stamp: "Formatted at 2026-08-10 23:35:47 on dist-test-slave-04bb"
server_key: "a01df406fde201552a333836f869d2bb"
server_key_iv: "5afb06f08cd3e98ca3ba0f97da81e99b"
server_key_version: "encryptionkey@0"
I20260810 23:36:33.775503 3696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:33.799458 3696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:33.804047 3696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:33.804365 3696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:33.825112 3696 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:44525
I20260810 23:36:33.825125 3785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:44525 every 8 connection(s)
I20260810 23:36:33.826704 3696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:36:33.829419 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 3696
I20260810 23:36:33.830505 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:46583
--local_ip_for_outbound_sockets=127.9.48.193
--tserver_master_addrs=127.9.48.254:44525
--webserver_port=35867
--webserver_interface=127.9.48.193
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:36:33.836710 3786 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:36:33.842164 3786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Bootstrap starting.
I20260810 23:36:33.864029 3786 log.cc:824] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Log is configured to *not* fsync() on all Append() calls
I20260810 23:36:33.881268 3786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:36:33.881989 3786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Bootstrap complete.
I20260810 23:36:33.891709 3786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } }
I20260810 23:36:33.892740 3786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184559be7e544fbeb0b19ac9f7e24f75, State: Initialized, Role: FOLLOWER
I20260810 23:36:33.893425 3786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } }
I20260810 23:36:33.893668 3786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:36:33.893787 3786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:36:33.893989 3786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:36:33.898092 3786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } }
I20260810 23:36:33.898619 3786 leader_election.cc:302] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 184559be7e544fbeb0b19ac9f7e24f75; no voters:
I20260810 23:36:33.899361 3786 leader_election.cc:288] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 23:36:33.899510 3794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:36:33.900818 3794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [term 3 LEADER]: Becoming Leader. State: Replica: 184559be7e544fbeb0b19ac9f7e24f75, State: Running, Role: LEADER
I20260810 23:36:33.901454 3794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } }
I20260810 23:36:33.901722 3786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:36:33.904310 3798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 184559be7e544fbeb0b19ac9f7e24f75. Latest consensus state: current_term: 3 leader_uuid: "184559be7e544fbeb0b19ac9f7e24f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } } }
I20260810 23:36:33.904772 3798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [sys.catalog]: This master's current role is: LEADER
I20260810 23:36:33.904333 3797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "184559be7e544fbeb0b19ac9f7e24f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "184559be7e544fbeb0b19ac9f7e24f75" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44525 } } }
I20260810 23:36:33.904932 3797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75 [sys.catalog]: This master's current role is: LEADER
I20260810 23:36:33.914427 3807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:36:33.925531 3807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:36:33.927835 3807 catalog_manager.cc:1259] Loaded cluster ID: 1ffbf61718644dabbf84140370f53026
I20260810 23:36:33.928040 3807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:36:33.933714 3807 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:36:33.936156 3807 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Loaded TSK: 0
I20260810 23:36:33.936440 3807 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 184559be7e544fbeb0b19ac9f7e24f75: Loaded TSK: 1
F20260810 23:36:33.937796 3807 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3291b989ad google::LogMessage::Fail() at ??:0
@ 0x7f3291b9b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f3291b984de google::LogMessage::Flush() at ??:0
@ 0x7f3291b99a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f32958fc7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f32958eb73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f32a75cb1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f32a75ca783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f32a76562a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f32a76560c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f32a76a05e9 std::function<>::operator()() at ??:0
@ 0x7f32a75cecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f32a75ccc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f32a7654610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f32a7654439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f32947e4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f32947e791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f32947e7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f32947c2eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3297a9c6db start_thread at ??:0
@ 0x7f3290a3261f clone at ??:0
W20260810 23:36:34.266593 3791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:34.266888 3791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:34.267067 3791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:34.274427 3791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:34.274533 3791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:34.274703 3791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:36:34.285086 3791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:46583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=35867
--tserver_master_addrs=127.9.48.254:44525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:34.286428 3791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:34.287778 3791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:34.301396 3862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:34.301355 3863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:34.301438 3865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:34.302491 3791 server_base.cc:1034] running on GCE node
I20260810 23:36:34.303108 3791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:34.304013 3791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:34.305244 3791 hybrid_clock.cc:648] HybridClock initialized: now 1786404994305192 us; error 51 us; skew 500 ppm
I20260810 23:36:34.305572 3791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:36:34 dist-test-slave-04bb krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404994, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:36:34.318339 3791 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:36:34.319898 3791 webserver.cc:460] Webserver started at http://127.9.48.193:35867/ using document root <none> and password file <none>
I20260810 23:36:34.320699 3791 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:34.320863 3791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:34.326273 3791 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:36:34.329190 3873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:34.330556 3791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:36:34.330741 3791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "837e557a1dde466b948d188ee833077b"
format_stamp: "Formatted at 2026-08-10 23:35:48 on dist-test-slave-04bb"
server_key: "f4037bb04a904f875770deaa19634779"
server_key_iv: "96d6c1e128935fec4fcb3519ba6494e2"
server_key_version: "encryptionkey@0"
I20260810 23:36:34.331723 3791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:34.346410 3791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:34.350455 3791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:34.350739 3791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:34.351688 3791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:36:34.353585 3882 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260810 23:36:34.359422 3791 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:36:34.359696 3791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260810 23:36:34.359844 3791 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:36:34.363103 3791 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:36:34.363232 3791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260810 23:36:34.363582 3882 tablet_bootstrap.cc:492] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b: Bootstrap starting.
I20260810 23:36:34.391918 3882 log.cc:824] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:36:34.406980 3882 tablet_bootstrap.cc:492] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:36:34.407573 3882 tablet_bootstrap.cc:492] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b: Bootstrap complete.
I20260810 23:36:34.409252 3882 ts_tablet_manager.cc:1397] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b: Time spent bootstrapping tablet: real 0.046s user 0.025s sys 0.019s
I20260810 23:36:34.410090 3791 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:46583
I20260810 23:36:34.410162 4050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:46583 every 8 connection(s)
I20260810 23:36:34.411451 3791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:36:34.417233 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:34.414567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:37295)
0810 23:36:34.415545 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:34.415856 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":410,"threads_started":1}
I20260810 23:36:34.417289 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 3791
W20260810 23:36:34.418069 4052 heartbeater.cc:644] Failed to heartbeat to 127.9.48.254:44525 (0 consecutive failures): Network error: Failed to ping master at 127.9.48.254:44525: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
I20260810 23:36:34.418156 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:44837
--local_ip_for_outbound_sockets=127.9.48.194
--tserver_master_addrs=127.9.48.254:44525
--webserver_port=37693
--webserver_interface=127.9.48.194
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:36:34.419154 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:34.418724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:46425)
0810 23:36:34.418917 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:34.419009 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 23:36:34.418092 3882 raft_consensus.cc:348] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:34.419425 3882 raft_consensus.cc:729] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 837e557a1dde466b948d188ee833077b, State: Initialized, Role: FOLLOWER
I20260810 23:36:34.420253 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:34.419835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:57093)
0810 23:36:34.419981 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:34.420076 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 23:36:34.420150 3882 consensus_queue.cc:260] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
W20260810 23:36:34.420598 4052 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:36:34.421907 3882 ts_tablet_manager.cc:1428] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260810 23:36:34.421967 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:34.421564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:53903)
0810 23:36:34.421737 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:34.421824 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 23:36:34.422786 4057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:34.422453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:60005)
0810 23:36:34.422595 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:34.422669 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:36:34.767428 4063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:34.767722 4063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:34.767935 4063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:34.775208 4063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:34.775315 4063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:34.775498 4063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:36:34.785384 4063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:44837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=37693
--tserver_master_addrs=127.9.48.254:44525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:34.786777 4063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:34.788208 4063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:34.802595 4071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:34.802634 4076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:34.802672 4072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:34.803893 4063 server_base.cc:1034] running on GCE node
I20260810 23:36:34.804575 4063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:34.805568 4063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:34.806816 4063 hybrid_clock.cc:648] HybridClock initialized: now 1786404994806765 us; error 45 us; skew 500 ppm
I20260810 23:36:34.807164 4063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:36:34 dist-test-slave-04bb krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404994, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:36:34.810741 4063 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:36:34.812372 4063 webserver.cc:460] Webserver started at http://127.9.48.194:37693/ using document root <none> and password file <none>
I20260810 23:36:34.813150 4063 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:34.813303 4063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:34.818956 4063 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:36:34.822189 4089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:34.823661 4063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:36:34.823863 4063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "fa970edeeef3483e981d005986bd0951"
format_stamp: "Formatted at 2026-08-10 23:35:48 on dist-test-slave-04bb"
server_key: "ced851f212b35e5de7496a884cfcf256"
server_key_iv: "75693b5aa77aabb8ec46aa4bc54e162e"
server_key_version: "encryptionkey@0"
I20260810 23:36:34.825013 4063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:34.836922 4063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:34.841136 4063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:34.841435 4063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:34.842484 4063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:36:34.843844 4063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:36:34.843993 4063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:34.844118 4063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:36:34.844209 4063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:34.880086 4063 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:44837
I20260810 23:36:34.880192 4260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:44837 every 8 connection(s)
I20260810 23:36:34.881845 4063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:36:34.887243 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:34.885220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:55503)
0810 23:36:34.886028 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:34.886307 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":291,"threads_started":1}
W20260810 23:36:34.888055 4261 heartbeater.cc:644] Failed to heartbeat to 127.9.48.254:44525 (0 consecutive failures): Network error: Failed to ping master at 127.9.48.254:44525: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
I20260810 23:36:34.888926 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:34.888581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:51259)
0810 23:36:34.888736 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:34.888808 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 23:36:34.889853 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:34.889527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:57959)
0810 23:36:34.889655 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:34.889737 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 23:36:34.890134 4261 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:36:34.891259 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 4063
I20260810 23:36:34.891911 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:44079
--local_ip_for_outbound_sockets=127.9.48.195
--tserver_master_addrs=127.9.48.254:44525
--webserver_port=33069
--webserver_interface=127.9.48.195
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:36:35.246649 4266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:35.246942 4266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:35.247042 4266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:35.254272 4266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:35.254386 4266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:35.254583 4266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:36:35.264413 4266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:44079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=33069
--tserver_master_addrs=127.9.48.254:44525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:35.265820 4266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:35.267249 4266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:35.282182 4276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:35.282119 4277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:35.282207 4279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:35.283504 4266 server_base.cc:1034] running on GCE node
I20260810 23:36:35.284080 4266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:35.285058 4266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:35.286285 4266 hybrid_clock.cc:648] HybridClock initialized: now 1786404995286231 us; error 54 us; skew 500 ppm
I20260810 23:36:35.286612 4266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:36:35 dist-test-slave-04bb krb5kdc[2048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404995, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:36:35.289919 4266 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:36:35.291359 4266 webserver.cc:460] Webserver started at http://127.9.48.195:33069/ using document root <none> and password file <none>
I20260810 23:36:35.292006 4266 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:35.292109 4266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:35.297539 4266 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:36:35.300282 4288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:35.301710 4266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:36:35.301841 4266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "229f01207aec449cb3948ad9596890a6"
format_stamp: "Formatted at 2026-08-10 23:35:49 on dist-test-slave-04bb"
server_key: "ace0af1a573b449577c7561794a865c6"
server_key_iv: "e76d9bf63d249ce9f6aeb6cc88ce945d"
server_key_version: "encryptionkey@0"
I20260810 23:36:35.302802 4266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:35.320076 4266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:35.324736 4266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:35.324968 4266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:35.325977 4266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:36:35.327332 4266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:36:35.327422 4266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:35.327498 4266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:36:35.327555 4266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:35.363289 4266 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:44079
I20260810 23:36:35.363381 4469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:44079 every 8 connection(s)
I20260810 23:36:35.364848 4266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:36:35.370016 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:35.367935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:33287)
0810 23:36:35.368807 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:35.369100 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":317,"threads_started":1}
W20260810 23:36:35.370829 4470 heartbeater.cc:644] Failed to heartbeat to 127.9.48.254:44525 (0 consecutive failures): Network error: Failed to ping master at 127.9.48.254:44525: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
I20260810 23:36:35.371727 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:35.371359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:34069)
0810 23:36:35.371537 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:35.371616 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 23:36:35.372767 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:35.372347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:48529)
0810 23:36:35.372486 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:35.372632 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 23:36:35.373042 4470 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:36:35.374794 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 4266
I20260810 23:36:35.424353 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:35.423578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:35733)
0810 23:36:35.424059 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:35.424148 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":243,"threads_started":1}
I20260810 23:36:35.891507 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:35.890644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:42377)
0810 23:36:35.891164 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:35.891256 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":248,"threads_started":1}
I20260810 23:36:35.974622 4477 raft_consensus.cc:482] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:35.974939 4477 raft_consensus.cc:504] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:35.976677 4477 leader_election.cc:288] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079), fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837)
Aug 10 23:36:35 dist-test-slave-04bb krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404994, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
Aug 10 23:36:35 dist-test-slave-04bb krb5kdc[2048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786404994, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:36:36.004580 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:35.977462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:47727 (local address 127.9.48.194:44837)
0810 23:36:35.980078 (+ 2616us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:35.980093 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:35.980140 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:36:35.982669 (+ 2529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:35.982681 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:35.983120 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:35.983456 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:35.986472 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:35.988674 (+ 2202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:35.990650 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:35.991348 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:35.991569 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:35.995782 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:35.995790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:35.995831 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:35.995885 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:36.000477 (+ 4592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:36.001517 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:36.001523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:36.001528 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:36.001677 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:36.002205 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:36.002213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:36.002216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:36.002773 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:36.002973 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:36.003759 (+ 786us) server_negotiation.cc:299] Negotiation successful
0810 23:36:36.004139 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2418,"thread_start_us":2230,"threads_started":1}
I20260810 23:36:36.004691 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:35.977721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:60283 (local address 127.9.48.195:44079)
0810 23:36:35.979652 (+ 1931us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:35.979671 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:35.979719 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:36:35.982298 (+ 2579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:35.982314 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:35.982922 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:35.983368 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:35.986472 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:35.988684 (+ 2212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:35.990670 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:35.991329 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:35.991506 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:35.996034 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:35.996046 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:35.996064 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:35.996165 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:36.000453 (+ 4288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:36.001478 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:36.001489 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:36.001498 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:36.001707 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:36.002204 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:36.002211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:36.002215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:36.002923 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:36.003138 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:36.003754 (+ 616us) server_negotiation.cc:299] Negotiation successful
0810 23:36:36.004190 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1730,"thread_start_us":1615,"threads_started":1}
I20260810 23:36:36.005084 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:35.976946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:44837 (local address 127.9.48.193:47727)
0810 23:36:35.977736 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:35.977778 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:35.978770 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:35.983523 (+ 4753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:35.983535 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:35.984899 (+ 1364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:36:35.986199 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:35.986230 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:35.988908 (+ 2678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:35.988913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:35.990486 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:35.990499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:35.991591 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:35.991598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:35.991861 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:35.992794 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:36:35.992812 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:36:35.995418 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:36:36.000702 (+ 5284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:36.000712 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:36.000724 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:36.001347 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:36.001839 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:36.001845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:36.001849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:36.002078 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:36.002926 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:36:36.002934 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:36:36.003238 (+ 304us) client_negotiation.cc:769] Sending connection context
0810 23:36:36.003526 (+ 288us) client_negotiation.cc:240] Negotiation successful
0810 23:36:36.004310 (+ 784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":472,"spinlock_wait_cycles":6400,"thread_start_us":495,"threads_started":1}
I20260810 23:36:36.005108 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:35.977032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:44079 (local address 127.9.48.193:60283)
0810 23:36:35.978597 (+ 1565us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:35.978621 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:35.978802 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:35.983523 (+ 4721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:35.983535 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:35.984899 (+ 1364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:36:35.986200 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:35.986230 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:35.988967 (+ 2737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:35.988975 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:35.990490 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:35.990505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:35.991559 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:35.991572 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:35.991886 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:35.992723 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:36:35.992752 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:36:35.995754 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:36:36.000700 (+ 4946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:36.000708 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:36.000723 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:36.001331 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:36.001859 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:36.001862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:36.001864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:36.002077 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:36.003076 (+ 999us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:36:36.003081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:36:36.003247 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 23:36:36.003527 (+ 280us) client_negotiation.cc:240] Negotiation successful
0810 23:36:36.004128 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1399,"mutex_wait_us":231,"spinlock_wait_cycles":34816,"thread_start_us":1115,"threads_started":1}
W20260810 23:36:36.008394 3877 leader_election.cc:341] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079): Not found: Tablet not found: 1ac246863fef4d2a8da2a114f0281d68
W20260810 23:36:36.008700 3875 leader_election.cc:341] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837): Not found: Tablet not found: 1ac246863fef4d2a8da2a114f0281d68
I20260810 23:36:36.008872 3875 leader_election.cc:302] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 837e557a1dde466b948d188ee833077b; no voters: 229f01207aec449cb3948ad9596890a6, fa970edeeef3483e981d005986bd0951
I20260810 23:36:36.009328 4477 raft_consensus.cc:2726] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:36:36.374631 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:36.373651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:42725)
0810 23:36:36.374244 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:36.374375 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":338,"threads_started":1}
I20260810 23:36:36.425876 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:36.425345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:35839)
0810 23:36:36.425562 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:36.425707 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 23:36:36.893601 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:36.892699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:36459)
0810 23:36:36.893246 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:36.893343 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":255,"threads_started":1}
I20260810 23:36:37.376205 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:37.375548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:44733)
0810 23:36:37.375983 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:37.376059 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":249,"threads_started":1}
I20260810 23:36:37.427667 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:37.426835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:34563)
0810 23:36:37.427383 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:37.427467 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":241,"threads_started":1}
I20260810 23:36:37.895442 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:37.894624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:47701)
0810 23:36:37.895165 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:37.895257 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":273,"threads_started":1}
I20260810 23:36:37.977093 4495 raft_consensus.cc:482] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:37.977276 4495 raft_consensus.cc:504] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:37.977993 4495 leader_election.cc:288] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079), fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837)
W20260810 23:36:37.979056 3877 leader_election.cc:341] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079): Not found: Tablet not found: 1ac246863fef4d2a8da2a114f0281d68
W20260810 23:36:37.979243 3875 leader_election.cc:341] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837): Not found: Tablet not found: 1ac246863fef4d2a8da2a114f0281d68
I20260810 23:36:37.979415 3875 leader_election.cc:302] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 837e557a1dde466b948d188ee833077b; no voters: 229f01207aec449cb3948ad9596890a6, fa970edeeef3483e981d005986bd0951
I20260810 23:36:37.979751 4495 raft_consensus.cc:2726] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:36:38.377681 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:38.377014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:39517)
0810 23:36:38.377447 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:38.377523 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":235,"threads_started":1}
I20260810 23:36:38.429636 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:38.428645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:46567)
0810 23:36:38.429257 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:38.429419 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":318,"threads_started":1}
I20260810 23:36:38.897253 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:38.896447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:48863)
0810 23:36:38.896960 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:38.897050 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":273,"threads_started":1}
I20260810 23:36:39.379345 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:39.378548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:46987)
0810 23:36:39.379058 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:39.379153 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":255,"threads_started":1}
I20260810 23:36:39.431528 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:39.430770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:45905)
0810 23:36:39.431241 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:39.431331 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":241,"threads_started":1}
I20260810 23:36:39.899121 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:39.898352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:45257)
0810 23:36:39.898845 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:39.898934 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":253,"threads_started":1}
I20260810 23:36:40.116703 4508 raft_consensus.cc:482] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:40.116904 4508 raft_consensus.cc:504] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:40.117668 4508 leader_election.cc:288] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079), fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837)
W20260810 23:36:40.118769 3877 leader_election.cc:341] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079): Not found: Tablet not found: 1ac246863fef4d2a8da2a114f0281d68
W20260810 23:36:40.118978 3875 leader_election.cc:341] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837): Not found: Tablet not found: 1ac246863fef4d2a8da2a114f0281d68
I20260810 23:36:40.119123 3875 leader_election.cc:302] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 837e557a1dde466b948d188ee833077b; no voters: 229f01207aec449cb3948ad9596890a6, fa970edeeef3483e981d005986bd0951
I20260810 23:36:40.119400 4508 raft_consensus.cc:2726] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:36:40.381173 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:40.380366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:45377)
0810 23:36:40.380904 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:40.380985 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":294,"threads_started":1}
I20260810 23:36:40.433503 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:40.432592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:46865)
0810 23:36:40.433191 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:40.433292 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":296,"threads_started":1}
I20260810 23:36:40.901077 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:40.900111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:33809)
0810 23:36:40.900738 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:40.900867 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":349,"threads_started":1}
I20260810 23:36:41.382970 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:41.382085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:55373)
0810 23:36:41.382640 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:41.382740 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":345,"threads_started":1}
I20260810 23:36:41.435218 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:41.434454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:58349)
0810 23:36:41.434921 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:41.435014 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":258,"threads_started":1}
I20260810 23:36:41.902947 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:41.902027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:35009)
0810 23:36:41.902630 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:41.902736 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":284,"threads_started":1}
I20260810 23:36:42.384944 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:42.383982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:41571)
0810 23:36:42.384601 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:42.384703 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":262,"threads_started":1}
I20260810 23:36:42.437127 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:42.436241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:52413)
0810 23:36:42.436868 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:42.436955 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":324,"threads_started":1}
I20260810 23:36:42.904772 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:42.903907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:38267)
0810 23:36:42.904372 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:42.904468 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":248,"threads_started":1}
I20260810 23:36:43.187969 4525 raft_consensus.cc:482] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:43.188277 4525 raft_consensus.cc:504] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:43.189064 4525 leader_election.cc:288] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079), fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837)
W20260810 23:36:43.190040 3877 leader_election.cc:341] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079): Not found: Tablet not found: 1ac246863fef4d2a8da2a114f0281d68
W20260810 23:36:43.190310 3875 leader_election.cc:341] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837): Not found: Tablet not found: 1ac246863fef4d2a8da2a114f0281d68
I20260810 23:36:43.190451 3875 leader_election.cc:302] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 837e557a1dde466b948d188ee833077b; no voters: 229f01207aec449cb3948ad9596890a6, fa970edeeef3483e981d005986bd0951
I20260810 23:36:43.190747 4525 raft_consensus.cc:2726] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:36:43.386716 4527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:43.385904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:38485)
0810 23:36:43.386348 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:43.386504 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":239,"threads_started":1}
I20260810 23:36:43.439167 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:43.438138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:49175)
0810 23:36:43.438773 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:43.438892 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":331,"threads_started":1}
I20260810 23:36:43.907009 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:43.905897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:51683)
0810 23:36:43.906604 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:43.906744 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":325,"threads_started":1}
I20260810 23:36:44.388590 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:44.387727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:35149)
0810 23:36:44.388301 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:44.388380 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":310,"threads_started":1}
I20260810 23:36:44.441187 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:44.440324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:47141)
0810 23:36:44.440890 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:44.440975 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":334,"threads_started":1}
I20260810 23:36:44.909158 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:44.908135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:33823)
0810 23:36:44.908816 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:44.908906 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":367,"threads_started":1}
I20260810 23:36:45.390188 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:45.389480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:59273)
0810 23:36:45.389927 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:45.390007 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":245,"threads_started":1}
I20260810 23:36:45.438122 4539 raft_consensus.cc:482] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:45.438350 4539 raft_consensus.cc:504] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:45.439066 4539 leader_election.cc:288] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079), fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837)
W20260810 23:36:45.440083 3875 leader_election.cc:341] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837): Not found: Tablet not found: 1ac246863fef4d2a8da2a114f0281d68
W20260810 23:36:45.440284 3877 leader_election.cc:341] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079): Not found: Tablet not found: 1ac246863fef4d2a8da2a114f0281d68
I20260810 23:36:45.440430 3877 leader_election.cc:302] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 837e557a1dde466b948d188ee833077b; no voters: 229f01207aec449cb3948ad9596890a6, fa970edeeef3483e981d005986bd0951
I20260810 23:36:45.440714 4539 raft_consensus.cc:2726] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:36:45.442937 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:45.442064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:48411)
0810 23:36:45.442505 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:45.442624 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":247,"threads_started":1}
I20260810 23:36:45.911021 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:45.910145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:56771)
0810 23:36:45.910739 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:45.910829 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":276,"threads_started":1}
I20260810 23:36:46.392135 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:46.391222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:48439)
0810 23:36:46.391786 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:46.391946 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":259,"threads_started":1}
I20260810 23:36:46.444803 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:46.443894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:55467)
0810 23:36:46.444418 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:46.444620 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":291,"threads_started":1}
I20260810 23:36:46.912958 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:46.912069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:53575)
0810 23:36:46.912671 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:46.912772 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":301,"threads_started":1}
I20260810 23:36:47.393839 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:47.393132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:40375)
0810 23:36:47.393588 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:47.393666 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":247,"threads_started":1}
I20260810 23:36:47.446686 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:47.445846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:47855)
0810 23:36:47.446424 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:47.446508 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":282,"threads_started":1}
I20260810 23:36:47.914798 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:47.913945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:50485)
0810 23:36:47.914515 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:47.914620 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":318,"threads_started":1}
I20260810 23:36:48.395633 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:48.394782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:39301)
0810 23:36:48.395346 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:48.395458 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":275,"threads_started":1}
I20260810 23:36:48.448459 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:48.447668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:38131)
0810 23:36:48.448153 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:48.448242 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":265,"threads_started":1}
I20260810 23:36:48.916616 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:48.915778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:39637)
0810 23:36:48.916285 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:48.916378 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":260,"threads_started":1}
I20260810 23:36:49.397451 4555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:49.396682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:37343)
0810 23:36:49.397150 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:49.397249 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":227,"threads_started":1}
I20260810 23:36:49.450624 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:49.449745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.193:33369)
0810 23:36:49.450263 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:49.450354 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":256,"threads_started":1}
I20260810 23:36:49.918768 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:49.917984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.194:51523)
0810 23:36:49.918501 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:49.918595 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":285,"threads_started":1}
I20260810 23:36:50.165099 4560 raft_consensus.cc:482] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:50.165289 4560 raft_consensus.cc:504] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "229f01207aec449cb3948ad9596890a6" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 44079 } } peers { permanent_uuid: "fa970edeeef3483e981d005986bd0951" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 44837 } } peers { permanent_uuid: "837e557a1dde466b948d188ee833077b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 46583 } }
I20260810 23:36:50.165987 4560 leader_election.cc:288] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079), fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837)
W20260810 23:36:50.167019 3877 leader_election.cc:341] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 229f01207aec449cb3948ad9596890a6 (127.9.48.195:44079): Not found: Tablet not found: 1ac246863fef4d2a8da2a114f0281d68
W20260810 23:36:50.167253 3875 leader_election.cc:341] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa970edeeef3483e981d005986bd0951 (127.9.48.194:44837): Not found: Tablet not found: 1ac246863fef4d2a8da2a114f0281d68
I20260810 23:36:50.167405 3875 leader_election.cc:302] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 837e557a1dde466b948d188ee833077b; no voters: 229f01207aec449cb3948ad9596890a6, fa970edeeef3483e981d005986bd0951
I20260810 23:36:50.167685 4560 raft_consensus.cc:2726] T 1ac246863fef4d2a8da2a114f0281d68 P 837e557a1dde466b948d188ee833077b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:36:50.387086 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 3791
I20260810 23:36:50.399643 4565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:50.398657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:44525 (local address 127.9.48.195:54641)
0810 23:36:50.399123 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:50.399396 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.254:44525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":251,"threads_started":1}
I20260810 23:36:50.403785 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:50.388209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.9.48.193:46583)
0810 23:36:50.389344 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:50.389355 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:50.389391 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:36:50.389477 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:50.389490 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:50.389743 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:50.389881 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:50.391460 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:50.392483 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:50.393922 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:50.394475 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:50.394571 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:50.397357 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:50.397368 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:50.397382 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:50.397418 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:50.401132 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:50.401834 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:50.401841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:50.401844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:50.401980 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:50.402356 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:50.402359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:50.402361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:50.402782 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:50.403021 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:50.403301 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 23:36:50.403512 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":696,"threads_started":1}
I20260810 23:36:50.405143 4045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:36:50.495365 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 3791
I20260810 23:36:50.520388 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 4063
I20260810 23:36:50.535197 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:50.521290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.9.48.194:44837)
0810 23:36:50.521778 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:50.521782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:50.521799 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:36:50.521857 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:50.521861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:50.521922 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:50.522016 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:50.523429 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:50.524201 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:50.525833 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:50.526284 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:50.526396 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:50.529753 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:50.529759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:50.529762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:50.529797 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:50.532686 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:50.533473 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:50.533477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:50.533479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:50.533604 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:50.533944 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:50.533947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:50.533949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:50.534214 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:50.534410 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:50.534730 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 23:36:50.534897 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":272,"threads_started":1}
I20260810 23:36:50.535920 4252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:36:50.619879 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 4063
I20260810 23:36:50.647228 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 4266
I20260810 23:36:50.661773 4571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:50.648247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.9.48.195:44079)
0810 23:36:50.648884 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:50.648889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:50.648906 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:36:50.648963 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:50.648966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:50.649032 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:50.649161 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:50.650656 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:50.651461 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:50.652795 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:50.653226 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:50.653391 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:50.656015 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:50.656020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:50.656024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:50.656060 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:50.658829 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:50.659607 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:50.659614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:50.659617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:50.659765 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:50.660147 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:50.660152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:50.660155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:50.660451 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:50.660776 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:50.661134 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 23:36:50.661361 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":354,"threads_started":1}
I20260810 23:36:50.662657 4461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:36:50.748572 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 4266
2026-08-10T23:36:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65439 ms)
[----------] 30 tests from SecurityITest (312139 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:36:50 dist-test-slave-04bb krb5kdc[4575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:36:50 dist-test-slave-04bb krb5kdc[4575](info): set up 2 sockets
Aug 10 23:36:50 dist-test-slave-04bb krb5kdc[4575](info): commencing operation
krb5kdc: starting...
W20260810 23:36:52.134788 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.333s user 0.005s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:36:52 dist-test-slave-04bb krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:36:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:36:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:36:52.452680 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:34985
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:34985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:36:52.800302 4592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:52.800654 4592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:52.800848 4592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:52.808378 4592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:36:52.808507 4592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:52.808612 4592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:52.808672 4592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:36:52.808720 4592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:36:52.819099 4592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:34985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:34985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:52.820434 4592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:52.821818 4592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:52.834086 4598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:52.834136 4603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:52.834229 4600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:52.835769 4592 server_base.cc:1034] running on GCE node
I20260810 23:36:52.836694 4592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:52.838232 4592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:52.839481 4592 hybrid_clock.cc:648] HybridClock initialized: now 1786405012839444 us; error 48 us; skew 500 ppm
I20260810 23:36:52.839869 4592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:36:52 dist-test-slave-04bb krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405012, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:36:52.843096 4592 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:36:52.844729 4592 webserver.cc:460] Webserver started at http://127.9.48.254:46575/ using document root <none> and password file <none>
I20260810 23:36:52.845440 4592 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:52.845602 4592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:52.845939 4592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:36:52.849102 4592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "2b0c81ab46ca41569f978ce9fa2e35b4"
format_stamp: "Formatted at 2026-08-10 23:36:52 on dist-test-slave-04bb"
server_key: "8903d729081eee95c38fb06b6299e6b6"
server_key_iv: "041aabd5f27131449e982194558e3c8f"
server_key_version: "encryptionkey@0"
I20260810 23:36:52.849941 4592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "2b0c81ab46ca41569f978ce9fa2e35b4"
format_stamp: "Formatted at 2026-08-10 23:36:52 on dist-test-slave-04bb"
server_key: "8903d729081eee95c38fb06b6299e6b6"
server_key_iv: "041aabd5f27131449e982194558e3c8f"
server_key_version: "encryptionkey@0"
I20260810 23:36:52.855356 4592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:36:52.858919 4613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:52.860198 4592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:36:52.860399 4592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "2b0c81ab46ca41569f978ce9fa2e35b4"
format_stamp: "Formatted at 2026-08-10 23:36:52 on dist-test-slave-04bb"
server_key: "8903d729081eee95c38fb06b6299e6b6"
server_key_iv: "041aabd5f27131449e982194558e3c8f"
server_key_version: "encryptionkey@0"
I20260810 23:36:52.860658 4592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:52.884822 4592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:52.889333 4592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:52.889650 4592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:52.908571 4592 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:34985
I20260810 23:36:52.908667 4691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:34985 every 8 connection(s)
I20260810 23:36:52.910041 4592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:36:52.913933 4693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:36:52.914988 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 4592
I20260810 23:36:52.915257 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:36:52.915956 9411 external_mini_cluster.cc:1428] Setting key a329fd032234c4bfe9a59a4148b3cc9c
I20260810 23:36:52.927750 4693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4: Bootstrap starting.
I20260810 23:36:52.931543 4693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4: Neither blocks nor log segments found. Creating new log.
I20260810 23:36:52.932655 4693 log.cc:824] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:36:52 dist-test-slave-04bb krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:36:52.935913 4693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4: No bootstrap required, opened a new log
I20260810 23:36:52.943842 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:52.918449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.9.48.254:34985)
0810 23:36:52.919529 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:52.919547 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:52.919941 (+ 394us) server_negotiation.cc:407] Connection header received
0810 23:36:52.922379 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:52.922409 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:52.922916 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:52.923393 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:52.925338 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:52.927315 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:52.928846 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:52.929985 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:52.930193 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:52.935172 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:52.935182 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:52.935197 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:52.935235 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:52.938708 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:52.939724 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:52.939733 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:52.939746 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:52.939953 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:52.940579 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:52.940586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:52.940593 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:52.941538 (+ 945us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:52.941797 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:52.942321 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 23:36:52.942701 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":477,"threads_started":1}
I20260810 23:36:52.945596 4693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0c81ab46ca41569f978ce9fa2e35b4" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 34985 } }
I20260810 23:36:52.946008 4693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:36:52.946137 4693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0c81ab46ca41569f978ce9fa2e35b4, State: Initialized, Role: FOLLOWER
I20260810 23:36:52.946727 4693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0c81ab46ca41569f978ce9fa2e35b4" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 34985 } }
I20260810 23:36:52.947001 4693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:36:52.947155 4693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:36:52.947376 4693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:36:52.949841 4693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0c81ab46ca41569f978ce9fa2e35b4" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 34985 } }
I20260810 23:36:52.950500 4693 leader_election.cc:302] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b0c81ab46ca41569f978ce9fa2e35b4; no voters:
I20260810 23:36:52.951188 4693 leader_election.cc:288] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:36:52.951365 4700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:36:52.952169 4700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [term 1 LEADER]: Becoming Leader. State: Replica: 2b0c81ab46ca41569f978ce9fa2e35b4, State: Running, Role: LEADER
I20260810 23:36:52.952811 4700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0c81ab46ca41569f978ce9fa2e35b4" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 34985 } }
I20260810 23:36:52.953505 4693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:36:52.956352 4701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b0c81ab46ca41569f978ce9fa2e35b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0c81ab46ca41569f978ce9fa2e35b4" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 34985 } } }
I20260810 23:36:52.956995 4701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:36:52.956864 4704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b0c81ab46ca41569f978ce9fa2e35b4. Latest consensus state: current_term: 1 leader_uuid: "2b0c81ab46ca41569f978ce9fa2e35b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b0c81ab46ca41569f978ce9fa2e35b4" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 34985 } } }
I20260810 23:36:52.957207 4704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:36:52.960875 4714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:36:52.966244 4714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:36:52.976202 4714 catalog_manager.cc:1347] Generated new cluster ID: 189ac8de017c4a6ea9ab40749c413b65
I20260810 23:36:52.976375 4714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:36:52.983565 4714 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:36:52.984900 4714 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:36:52.990942 4714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4: Generated new TSK 0
I20260810 23:36:52.991860 4714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:36:53.073103 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:34985
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:36:53.419418 4728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:53.419747 4728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:53.419927 4728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:53.426895 4728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:53.427044 4728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:53.427251 4728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:36:53.437168 4728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:34985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:53.438486 4728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:53.439847 4728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:53.453366 4738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:53.453415 4741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:53.453588 4739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:53.454270 4728 server_base.cc:1034] running on GCE node
I20260810 23:36:53.454965 4728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:53.455919 4728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:53.457185 4728 hybrid_clock.cc:648] HybridClock initialized: now 1786405013457113 us; error 60 us; skew 500 ppm
I20260810 23:36:53.457495 4728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:36:53 dist-test-slave-04bb krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405013, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:36:53.460915 4728 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:36:53.462419 4728 webserver.cc:460] Webserver started at http://127.9.48.193:42619/ using document root <none> and password file <none>
I20260810 23:36:53.463063 4728 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:53.463176 4728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:53.463442 4728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:36:53.466454 4728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "3c54206c5ec34327a6cf8e37f53de11b"
format_stamp: "Formatted at 2026-08-10 23:36:53 on dist-test-slave-04bb"
server_key: "2964b93e68dd6514f004736c7b733f6b"
server_key_iv: "62d6ebbd7b0ff267a78301d33f877f0e"
server_key_version: "encryptionkey@0"
I20260810 23:36:53.467200 4728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "3c54206c5ec34327a6cf8e37f53de11b"
format_stamp: "Formatted at 2026-08-10 23:36:53 on dist-test-slave-04bb"
server_key: "2964b93e68dd6514f004736c7b733f6b"
server_key_iv: "62d6ebbd7b0ff267a78301d33f877f0e"
server_key_version: "encryptionkey@0"
I20260810 23:36:53.472378 4728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 23:36:53.475813 4749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:53.477319 4728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:36:53.477494 4728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "3c54206c5ec34327a6cf8e37f53de11b"
format_stamp: "Formatted at 2026-08-10 23:36:53 on dist-test-slave-04bb"
server_key: "2964b93e68dd6514f004736c7b733f6b"
server_key_iv: "62d6ebbd7b0ff267a78301d33f877f0e"
server_key_version: "encryptionkey@0"
I20260810 23:36:53.477692 4728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:53.498898 4728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:53.502988 4728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:53.503275 4728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:53.504236 4728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:36:53.505646 4728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:36:53.505731 4728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:53.505895 4728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:36:53.505957 4728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:53.540062 4728 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:32933
I20260810 23:36:53.540190 4920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:32933 every 8 connection(s)
I20260810 23:36:53.541457 4728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:36:53.546000 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 4728
I20260810 23:36:53.546243 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:36:53.546912 9411 external_mini_cluster.cc:1428] Setting key 034e931442f74f3eda2e594651591541
Aug 10 23:36:53 dist-test-slave-04bb krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405013, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:36:53.565280 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:53.544176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:52241 (local address 127.9.48.254:34985)
0810 23:36:53.544736 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:53.544741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:53.545720 (+ 979us) server_negotiation.cc:407] Connection header received
0810 23:36:53.547865 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:53.547871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:53.547962 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:53.548091 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:53.549978 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:53.551175 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:53.553078 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:53.553830 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:53.553995 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:53.558018 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:53.558028 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:53.558035 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:53.558082 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:53.561785 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:53.562573 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:53.562581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:53.562587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:53.562750 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:53.563160 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:53.563168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:53.563174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:53.563540 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:53.563761 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:53.564657 (+ 896us) server_negotiation.cc:299] Negotiation successful
0810 23:36:53.564939 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":245,"threads_started":1}
I20260810 23:36:53.566593 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:53.544778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:34985 (local address 127.9.48.193:52241)
0810 23:36:53.545541 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:53.545580 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:53.547522 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:53.548372 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:53.548383 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:53.549170 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:36:53.549782 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:53.549804 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:53.551391 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:53.551396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:53.552922 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:53.552936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:53.554173 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:53.554180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:53.554469 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:53.555249 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:36:53.555272 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:36:53.557761 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:36:53.561935 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:53.561941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:53.561991 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:53.562437 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:53.562862 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:53.562865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:53.562867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:53.563039 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:53.563677 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:36:53.563688 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:36:53.564153 (+ 465us) client_negotiation.cc:769] Sending connection context
0810 23:36:53.564478 (+ 325us) client_negotiation.cc:240] Negotiation successful
0810 23:36:53.565104 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":295,"threads_started":1}
I20260810 23:36:53.568279 4922 heartbeater.cc:347] Connected to a master server at 127.9.48.254:34985
I20260810 23:36:53.568658 4922 heartbeater.cc:464] Registering TS with master...
I20260810 23:36:53.569407 4922 heartbeater.cc:511] Master 127.9.48.254:34985 requested a full tablet report, sending...
I20260810 23:36:53.571558 4634 ts_manager.cc:194] Registered new tserver with Master: 3c54206c5ec34327a6cf8e37f53de11b (127.9.48.193:32933)
I20260810 23:36:53.573123 4634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:52241
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:36:53.620185 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:34985
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:36:53.962184 4937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:53.962458 4937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:53.962644 4937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:53.969955 4937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:53.970060 4937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:53.970243 4937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:36:53.979889 4937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:34985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:53.981284 4937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:53.982833 4937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:53.996498 4946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:53.996502 4951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:53.996551 4947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:54.002605 4937 server_base.cc:1034] running on GCE node
I20260810 23:36:54.003340 4937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:54.004355 4937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:54.005630 4937 hybrid_clock.cc:648] HybridClock initialized: now 1786405014005577 us; error 59 us; skew 500 ppm
I20260810 23:36:54.005972 4937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:36:54 dist-test-slave-04bb krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405014, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:36:54.009421 4937 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:36:54.011081 4937 webserver.cc:460] Webserver started at http://127.9.48.194:39075/ using document root <none> and password file <none>
I20260810 23:36:54.011938 4937 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:54.012104 4937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:54.012442 4937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:36:54.015460 4937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "93262860f53b457aabd5a837aae4c392"
format_stamp: "Formatted at 2026-08-10 23:36:54 on dist-test-slave-04bb"
server_key: "fecd5f753645687a0d16abd9d981a194"
server_key_iv: "3f343b4c58faf21ed5b652ad13569818"
server_key_version: "encryptionkey@0"
I20260810 23:36:54.016260 4937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "93262860f53b457aabd5a837aae4c392"
format_stamp: "Formatted at 2026-08-10 23:36:54 on dist-test-slave-04bb"
server_key: "fecd5f753645687a0d16abd9d981a194"
server_key_iv: "3f343b4c58faf21ed5b652ad13569818"
server_key_version: "encryptionkey@0"
I20260810 23:36:54.022207 4937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:36:54.025949 4960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:54.027371 4937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:36:54.027576 4937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "93262860f53b457aabd5a837aae4c392"
format_stamp: "Formatted at 2026-08-10 23:36:54 on dist-test-slave-04bb"
server_key: "fecd5f753645687a0d16abd9d981a194"
server_key_iv: "3f343b4c58faf21ed5b652ad13569818"
server_key_version: "encryptionkey@0"
I20260810 23:36:54.027801 4937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:54.047924 4937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:54.052399 4937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:54.052757 4937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:54.053833 4937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:36:54.055266 4937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:36:54.055380 4937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:54.055536 4937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:36:54.055613 4937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:54.091777 4937 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:43767
I20260810 23:36:54.091796 5137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:43767 every 8 connection(s)
I20260810 23:36:54.094034 4937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:36:54.102634 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 4937
I20260810 23:36:54.102851 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:36:54.103413 9411 external_mini_cluster.cc:1428] Setting key d4e7755f1c6f4250273c81f3f3ab8bbe
Aug 10 23:36:54 dist-test-slave-04bb krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405014, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:36:54.137773 5147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.108300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:53331 (local address 127.9.48.254:34985)
0810 23:36:54.116990 (+ 8690us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:54.116995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:54.117017 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:36:54.117086 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:54.117091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:54.117165 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:54.117335 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:54.119296 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.120176 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.122090 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.122820 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.122991 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.131384 (+ 8393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:54.131390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:54.131394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:54.131430 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:54.134461 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:54.135248 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:54.135254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:54.135259 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:54.135403 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:54.135870 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:54.135880 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:54.135882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:54.136159 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:54.136330 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:54.137121 (+ 791us) server_negotiation.cc:299] Negotiation successful
0810 23:36:54.137357 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8565,"thread_start_us":8425,"threads_started":1}
I20260810 23:36:54.138811 5145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.107615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:34985 (local address 127.9.48.194:53331)
0810 23:36:54.108634 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:54.108688 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:54.110735 (+ 2047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:54.117654 (+ 6919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:54.117665 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:54.118363 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:36:54.119053 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:54.119083 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.120370 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.120383 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:54.121904 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:54.121919 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.123112 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.123117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:54.123344 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.128042 (+ 4698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:36:54.128084 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:36:54.131010 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:36:54.134631 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:54.134640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:54.134656 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:54.135093 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:54.135529 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:54.135534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:54.135538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:54.135736 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:54.136317 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:36:54.136325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:36:54.136738 (+ 413us) client_negotiation.cc:769] Sending connection context
0810 23:36:54.137018 (+ 280us) client_negotiation.cc:240] Negotiation successful
0810 23:36:54.137587 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":419,"threads_started":1}
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
I20260810 23:36:54.140489 5140 heartbeater.cc:347] Connected to a master server at 127.9.48.254:34985
I20260810 23:36:54.140874 5140 heartbeater.cc:464] Registering TS with master...
I20260810 23:36:54.141716 5140 heartbeater.cc:511] Master 127.9.48.254:34985 requested a full tablet report, sending...
I20260810 23:36:54.143268 4634 ts_manager.cc:194] Registered new tserver with Master: 93262860f53b457aabd5a837aae4c392 (127.9.48.194:43767)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
I20260810 23:36:54.144554 4634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:53331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:36:54.184720 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:34985
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:36:54.530493 5151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:54.530771 5151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:54.530934 5151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:54.538177 5151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:54.538280 5151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:54.538450 5151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:36:54.548051 5151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:34985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:54.549283 5151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:54.550560 5151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:54.564513 5162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:54.564757 5164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:54.564872 5167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:54.565804 5151 server_base.cc:1034] running on GCE node
I20260810 23:36:54.566480 5151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:54.567369 5151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:54.568653 5151 hybrid_clock.cc:648] HybridClock initialized: now 1786405014568613 us; error 35 us; skew 500 ppm
I20260810 23:36:54.568982 5151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:36:54 dist-test-slave-04bb krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405014, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:36:54.572181 5151 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:36:54.573745 5151 webserver.cc:460] Webserver started at http://127.9.48.195:46369/ using document root <none> and password file <none>
I20260810 23:36:54.574445 5151 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:54.574589 5151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:54.574915 5151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:36:54.576020 4922 heartbeater.cc:503] Master 127.9.48.254:34985 was elected leader, sending a full tablet report...
I20260810 23:36:54.578413 5151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "2587d021327147b29fe6d84d679e1ca3"
format_stamp: "Formatted at 2026-08-10 23:36:54 on dist-test-slave-04bb"
server_key: "63cfcec9916f3628b89781a5892ac71a"
server_key_iv: "051d7b6b140508ac4a5922a89551633d"
server_key_version: "encryptionkey@0"
I20260810 23:36:54.579207 5151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "2587d021327147b29fe6d84d679e1ca3"
format_stamp: "Formatted at 2026-08-10 23:36:54 on dist-test-slave-04bb"
server_key: "63cfcec9916f3628b89781a5892ac71a"
server_key_iv: "051d7b6b140508ac4a5922a89551633d"
server_key_version: "encryptionkey@0"
I20260810 23:36:54.584682 5151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 23:36:54.588068 5177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:54.589349 5151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:36:54.589536 5151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "2587d021327147b29fe6d84d679e1ca3"
format_stamp: "Formatted at 2026-08-10 23:36:54 on dist-test-slave-04bb"
server_key: "63cfcec9916f3628b89781a5892ac71a"
server_key_iv: "051d7b6b140508ac4a5922a89551633d"
server_key_version: "encryptionkey@0"
I20260810 23:36:54.589749 5151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:54.606364 5151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:54.611008 5151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:54.611294 5151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:54.612388 5151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:36:54.613886 5151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:36:54.613994 5151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:54.614135 5151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:36:54.614221 5151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:54.649863 5151 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:34885
I20260810 23:36:54.649885 5379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:34885 every 8 connection(s)
I20260810 23:36:54.651237 5151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:36:54.656340 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 5151
I20260810 23:36:54.656592 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:36:54.657097 9411 external_mini_cluster.cc:1428] Setting key 49e5e4e3bb451c0292bdab8fa300ed30
Aug 10 23:36:54 dist-test-slave-04bb krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405014, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:36:54.674264 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.654008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:55659 (local address 127.9.48.254:34985)
0810 23:36:54.654550 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:54.654555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:54.655704 (+ 1149us) server_negotiation.cc:407] Connection header received
0810 23:36:54.658519 (+ 2815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:54.658524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:54.658612 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:54.658742 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:54.661165 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.662367 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.664013 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.664428 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.664563 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.668313 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:54.668318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:54.668322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:54.668354 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:54.670888 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:54.671757 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:54.671762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:54.671771 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:54.671889 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:54.672357 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:54.672362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:54.672371 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:54.672795 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:54.672969 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:54.673660 (+ 691us) server_negotiation.cc:299] Negotiation successful
0810 23:36:54.673880 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":272,"threads_started":1}
I20260810 23:36:54.675211 5387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.654502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:34985 (local address 127.9.48.195:55659)
0810 23:36:54.655452 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:54.655504 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:54.658062 (+ 2558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:54.659068 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:54.659080 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:54.659886 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:36:54.660948 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:54.660981 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.662534 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.662538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:54.663898 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:54.663910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.664682 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.664688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:54.664940 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.665718 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:36:54.665744 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:36:54.668093 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:36:54.671032 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:54.671042 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:54.671058 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:54.671644 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:54.672007 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:36:54.672011 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:36:54.672015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:36:54.672222 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:36:54.672933 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:36:54.672942 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:36:54.673284 (+ 342us) client_negotiation.cc:769] Sending connection context
0810 23:36:54.673556 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 23:36:54.674091 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":404,"threads_started":1}
I20260810 23:36:54.676633 5380 heartbeater.cc:347] Connected to a master server at 127.9.48.254:34985
I20260810 23:36:54.676985 5380 heartbeater.cc:464] Registering TS with master...
I20260810 23:36:54.677773 5380 heartbeater.cc:511] Master 127.9.48.254:34985 requested a full tablet report, sending...
I20260810 23:36:54.679534 4634 ts_manager.cc:194] Registered new tserver with Master: 2587d021327147b29fe6d84d679e1ca3 (127.9.48.195:34885)
I20260810 23:36:54.680469 4634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:55659
I20260810 23:36:54.685406 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:36:54.703708 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.690535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58958 (local address 127.9.48.254:34985)
0810 23:36:54.690722 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:54.690727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:54.691213 (+ 486us) server_negotiation.cc:407] Connection header received
0810 23:36:54.691337 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:54.691340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:54.691415 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:54.691504 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:54.693238 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.694069 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.695497 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.695881 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.695977 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.698534 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:54.698539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:54.698543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:54.698578 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:54.701312 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:54.701968 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:54.701972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:54.701974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:54.702091 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:54.702451 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:54.702454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:54.702456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:54.702783 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:54.702969 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:54.703310 (+ 341us) server_negotiation.cc:299] Negotiation successful
0810 23:36:54.703475 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:36:54.707927 4634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:36:54.715126 4634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:36:54.740365 5404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.727618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:43767 (local address 127.0.0.1:60990)
0810 23:36:54.728750 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:54.728827 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:54.729054 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:54.731910 (+ 2856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:54.731917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:54.731947 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:36:54.732365 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:54.732380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.734431 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.734437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:54.737065 (+ 2628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:54.737081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.739196 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.739203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:54.739421 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.739485 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 23:36:54.739621 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 23:36:54.739820 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":53,"thread_start_us":639,"threads_started":1}
I20260810 23:36:54.740510 5408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.728388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53018 (local address 127.9.48.195:34885)
0810 23:36:54.730544 (+ 2156us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:54.730556 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:54.731510 (+ 954us) server_negotiation.cc:407] Connection header received
0810 23:36:54.731851 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:54.731860 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:54.732110 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:54.732292 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:36:54.733453 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.735021 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.737676 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.739088 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.739193 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.739409 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:54.740006 (+ 597us) server_negotiation.cc:299] Negotiation successful
0810 23:36:54.740170 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1937,"thread_start_us":1754,"threads_started":1}
I20260810 23:36:54.741123 5405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.727861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.9.48.194:43767)
0810 23:36:54.730969 (+ 3108us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:54.730977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:54.731006 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:36:54.731143 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:54.731154 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:54.731411 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:54.731632 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:36:54.732603 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.734267 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.737261 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.738722 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.738846 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.739037 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:54.740648 (+ 1611us) server_negotiation.cc:299] Negotiation successful
0810 23:36:54.740823 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2969,"thread_start_us":2860,"threads_started":1}
I20260810 23:36:54.741511 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.727973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:34885 (local address 127.0.0.1:53018)
0810 23:36:54.731393 (+ 3420us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:54.731416 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:54.731647 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:54.732725 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:54.732730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:54.732758 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:36:54.733236 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:54.733250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.735218 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.735223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:54.737479 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:54.737496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.739616 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.739623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:54.739796 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.739827 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 23:36:54.740988 (+ 1161us) client_negotiation.cc:240] Negotiation successful
0810 23:36:54.741129 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3303,"mutex_wait_us":63,"thread_start_us":3501,"threads_started":1}
I20260810 23:36:54.745218 5037 tablet_service.cc:1467] Processing CreateTablet for tablet c31ed8be62a04d949bbc6d217a0fa106 (DEFAULT_TABLE table=test-table [id=239acc13c1274de2b2b8f844cbcae7d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:36:54.745217 5265 tablet_service.cc:1467] Processing CreateTablet for tablet c31ed8be62a04d949bbc6d217a0fa106 (DEFAULT_TABLE table=test-table [id=239acc13c1274de2b2b8f844cbcae7d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:36:54.746747 5037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31ed8be62a04d949bbc6d217a0fa106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:36:54.746768 5265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31ed8be62a04d949bbc6d217a0fa106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:36:54.750535 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.730294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:32933 (local address 127.0.0.1:37552)
0810 23:36:54.740585 (+ 10291us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:54.740613 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:54.740743 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:54.743870 (+ 3127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:54.743878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:54.743915 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:36:54.744281 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:54.744296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.745872 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.745882 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:54.747880 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:54.747899 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.749536 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.749544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:54.749749 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.749800 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 23:36:54.749951 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 23:36:54.750111 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10114,"thread_start_us":9889,"threads_started":1}
I20260810 23:36:54.750591 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.731161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.9.48.193:32933)
0810 23:36:54.743289 (+ 12128us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:54.743300 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:54.743337 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:36:54.743462 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:54.743469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:54.743645 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:54.743905 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:36:54.744474 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.745697 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.748121 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.749281 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.749478 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.749757 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:54.750008 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 23:36:54.750183 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11944,"thread_start_us":11703,"threads_started":1}
I20260810 23:36:54.754649 4824 tablet_service.cc:1467] Processing CreateTablet for tablet c31ed8be62a04d949bbc6d217a0fa106 (DEFAULT_TABLE table=test-table [id=239acc13c1274de2b2b8f844cbcae7d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:36:54.756079 4824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31ed8be62a04d949bbc6d217a0fa106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:36:54.762755 5417 tablet_bootstrap.cc:492] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3: Bootstrap starting.
I20260810 23:36:54.763387 5416 tablet_bootstrap.cc:492] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392: Bootstrap starting.
I20260810 23:36:54.766562 5417 tablet_bootstrap.cc:654] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3: Neither blocks nor log segments found. Creating new log.
I20260810 23:36:54.767547 5416 tablet_bootstrap.cc:654] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392: Neither blocks nor log segments found. Creating new log.
I20260810 23:36:54.767617 5417 log.cc:824] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:36:54.769061 5416 log.cc:824] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392: Log is configured to *not* fsync() on all Append() calls
I20260810 23:36:54.771019 5417 tablet_bootstrap.cc:492] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3: No bootstrap required, opened a new log
I20260810 23:36:54.771320 5417 ts_tablet_manager.cc:1397] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260810 23:36:54.772917 5424 tablet_bootstrap.cc:492] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b: Bootstrap starting.
I20260810 23:36:54.773351 5416 tablet_bootstrap.cc:492] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392: No bootstrap required, opened a new log
I20260810 23:36:54.773756 5416 ts_tablet_manager.cc:1397] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260810 23:36:54.776508 5424 tablet_bootstrap.cc:654] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b: Neither blocks nor log segments found. Creating new log.
I20260810 23:36:54.777791 5424 log.cc:824] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:36:54.781563 5424 tablet_bootstrap.cc:492] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b: No bootstrap required, opened a new log
I20260810 23:36:54.781857 5424 ts_tablet_manager.cc:1397] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260810 23:36:54.781704 5417 raft_consensus.cc:348] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93262860f53b457aabd5a837aae4c392" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43767 } } peers { permanent_uuid: "2587d021327147b29fe6d84d679e1ca3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34885 } } peers { permanent_uuid: "3c54206c5ec34327a6cf8e37f53de11b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 32933 } }
I20260810 23:36:54.782172 5417 raft_consensus.cc:374] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:36:54.782280 5417 raft_consensus.cc:729] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2587d021327147b29fe6d84d679e1ca3, State: Initialized, Role: FOLLOWER
I20260810 23:36:54.782909 5417 consensus_queue.cc:260] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93262860f53b457aabd5a837aae4c392" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43767 } } peers { permanent_uuid: "2587d021327147b29fe6d84d679e1ca3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34885 } } peers { permanent_uuid: "3c54206c5ec34327a6cf8e37f53de11b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 32933 } }
I20260810 23:36:54.784153 5380 heartbeater.cc:503] Master 127.9.48.254:34985 was elected leader, sending a full tablet report...
I20260810 23:36:54.784004 5416 raft_consensus.cc:348] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93262860f53b457aabd5a837aae4c392" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43767 } } peers { permanent_uuid: "2587d021327147b29fe6d84d679e1ca3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34885 } } peers { permanent_uuid: "3c54206c5ec34327a6cf8e37f53de11b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 32933 } }
I20260810 23:36:54.784581 5416 raft_consensus.cc:374] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:36:54.784741 5416 raft_consensus.cc:729] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93262860f53b457aabd5a837aae4c392, State: Initialized, Role: FOLLOWER
I20260810 23:36:54.785645 5416 consensus_queue.cc:260] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93262860f53b457aabd5a837aae4c392" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43767 } } peers { permanent_uuid: "2587d021327147b29fe6d84d679e1ca3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34885 } } peers { permanent_uuid: "3c54206c5ec34327a6cf8e37f53de11b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 32933 } }
I20260810 23:36:54.786127 5417 ts_tablet_manager.cc:1428] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260810 23:36:54.787372 5140 heartbeater.cc:503] Master 127.9.48.254:34985 was elected leader, sending a full tablet report...
I20260810 23:36:54.788335 5416 ts_tablet_manager.cc:1428] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260810 23:36:54.792030 5424 raft_consensus.cc:348] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93262860f53b457aabd5a837aae4c392" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43767 } } peers { permanent_uuid: "2587d021327147b29fe6d84d679e1ca3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34885 } } peers { permanent_uuid: "3c54206c5ec34327a6cf8e37f53de11b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 32933 } }
I20260810 23:36:54.792490 5424 raft_consensus.cc:374] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:36:54.792647 5424 raft_consensus.cc:729] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c54206c5ec34327a6cf8e37f53de11b, State: Initialized, Role: FOLLOWER
I20260810 23:36:54.793299 5424 consensus_queue.cc:260] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93262860f53b457aabd5a837aae4c392" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43767 } } peers { permanent_uuid: "2587d021327147b29fe6d84d679e1ca3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34885 } } peers { permanent_uuid: "3c54206c5ec34327a6cf8e37f53de11b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 32933 } }
I20260810 23:36:54.795383 5424 ts_tablet_manager.cc:1428] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260810 23:36:54.855947 5143 tablet.cc:2369] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:36:54.867240 5428 raft_consensus.cc:482] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:36:54.867555 5428 raft_consensus.cc:504] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93262860f53b457aabd5a837aae4c392" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43767 } } peers { permanent_uuid: "2587d021327147b29fe6d84d679e1ca3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34885 } } peers { permanent_uuid: "3c54206c5ec34327a6cf8e37f53de11b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 32933 } }
I20260810 23:36:54.869591 5428 leader_election.cc:288] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93262860f53b457aabd5a837aae4c392 (127.9.48.194:43767), 2587d021327147b29fe6d84d679e1ca3 (127.9.48.195:34885)
I20260810 23:36:54.877008 5408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.870361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:50953 (local address 127.9.48.195:34885)
0810 23:36:54.870585 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:54.870591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:54.871319 (+ 728us) server_negotiation.cc:407] Connection header received
0810 23:36:54.871438 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:54.871444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:54.871535 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:54.871654 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:36:54.872315 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.873250 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.875111 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.875951 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.876166 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.876270 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:54.876554 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 23:36:54.876710 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:36:54.877028 5405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.869912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:53553 (local address 127.9.48.194:43767)
0810 23:36:54.870133 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:54.870141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:54.870865 (+ 724us) server_negotiation.cc:407] Connection header received
0810 23:36:54.871113 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:54.871119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:54.871227 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:54.871324 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:36:54.872073 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.872962 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.875210 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.875966 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.876163 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.876248 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:54.876664 (+ 416us) server_negotiation.cc:299] Negotiation successful
0810 23:36:54.876805 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:36:54.877097 5431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.870183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:34885 (local address 127.9.48.193:50953)
0810 23:36:54.871153 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:54.871176 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:54.871312 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:54.871687 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:54.871690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:54.871715 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:36:54.872103 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:54.872116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.873413 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.873417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:54.874924 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:54.874936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.876195 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.876202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:54.876359 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.876383 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:36:54.876459 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 23:36:54.876629 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":1,"thread_start_us":563,"threads_started":1}
I20260810 23:36:54.877238 5430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.869880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:43767 (local address 127.9.48.193:53553)
0810 23:36:54.870700 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:54.870734 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:54.870931 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:54.871374 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:54.871380 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:54.871425 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:36:54.871902 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:54.871915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.873138 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.873144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:54.874979 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:54.875023 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.876270 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.876278 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:54.876428 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.876492 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 23:36:54.876632 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 23:36:54.876756 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":82,"thread_start_us":567,"threads_started":1}
I20260810 23:36:54.878542 5066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31ed8be62a04d949bbc6d217a0fa106" candidate_uuid: "3c54206c5ec34327a6cf8e37f53de11b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93262860f53b457aabd5a837aae4c392" is_pre_election: true
I20260810 23:36:54.878542 5303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31ed8be62a04d949bbc6d217a0fa106" candidate_uuid: "3c54206c5ec34327a6cf8e37f53de11b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2587d021327147b29fe6d84d679e1ca3" is_pre_election: true
I20260810 23:36:54.879190 5066 raft_consensus.cc:2454] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c54206c5ec34327a6cf8e37f53de11b in term 0.
I20260810 23:36:54.879168 5303 raft_consensus.cc:2454] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c54206c5ec34327a6cf8e37f53de11b in term 0.
I20260810 23:36:54.879895 4752 leader_election.cc:302] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c54206c5ec34327a6cf8e37f53de11b, 93262860f53b457aabd5a837aae4c392; no voters:
I20260810 23:36:54.880347 5428 raft_consensus.cc:2781] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:36:54.880569 5428 raft_consensus.cc:482] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:36:54.880681 5428 raft_consensus.cc:3037] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:36:54.883206 5428 raft_consensus.cc:504] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93262860f53b457aabd5a837aae4c392" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43767 } } peers { permanent_uuid: "2587d021327147b29fe6d84d679e1ca3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34885 } } peers { permanent_uuid: "3c54206c5ec34327a6cf8e37f53de11b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 32933 } }
I20260810 23:36:54.883837 5428 leader_election.cc:288] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [CANDIDATE]: Term 1 election: Requested vote from peers 93262860f53b457aabd5a837aae4c392 (127.9.48.194:43767), 2587d021327147b29fe6d84d679e1ca3 (127.9.48.195:34885)
I20260810 23:36:54.884296 5066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31ed8be62a04d949bbc6d217a0fa106" candidate_uuid: "3c54206c5ec34327a6cf8e37f53de11b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93262860f53b457aabd5a837aae4c392"
I20260810 23:36:54.884392 5303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31ed8be62a04d949bbc6d217a0fa106" candidate_uuid: "3c54206c5ec34327a6cf8e37f53de11b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2587d021327147b29fe6d84d679e1ca3"
I20260810 23:36:54.884553 5066 raft_consensus.cc:3037] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:36:54.884665 5303 raft_consensus.cc:3037] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:36:54.886930 5066 raft_consensus.cc:2454] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c54206c5ec34327a6cf8e37f53de11b in term 1.
I20260810 23:36:54.887176 5303 raft_consensus.cc:2454] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c54206c5ec34327a6cf8e37f53de11b in term 1.
I20260810 23:36:54.887401 4752 leader_election.cc:302] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c54206c5ec34327a6cf8e37f53de11b, 93262860f53b457aabd5a837aae4c392; no voters:
I20260810 23:36:54.887719 5428 raft_consensus.cc:2781] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:36:54.888247 5428 raft_consensus.cc:686] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [term 1 LEADER]: Becoming Leader. State: Replica: 3c54206c5ec34327a6cf8e37f53de11b, State: Running, Role: LEADER
I20260810 23:36:54.888842 5428 consensus_queue.cc:237] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93262860f53b457aabd5a837aae4c392" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43767 } } peers { permanent_uuid: "2587d021327147b29fe6d84d679e1ca3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34885 } } peers { permanent_uuid: "3c54206c5ec34327a6cf8e37f53de11b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 32933 } }
I20260810 23:36:54.893450 4633 catalog_manager.cc:5495] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c54206c5ec34327a6cf8e37f53de11b (127.9.48.193). New cstate: current_term: 1 leader_uuid: "3c54206c5ec34327a6cf8e37f53de11b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93262860f53b457aabd5a837aae4c392" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2587d021327147b29fe6d84d679e1ca3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c54206c5ec34327a6cf8e37f53de11b" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 32933 } health_report { overall_health: HEALTHY } } }
W20260810 23:36:54.902894 5381 tablet.cc:2369] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:36:54.934666 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.929151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.9.48.193:32933)
0810 23:36:54.929374 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:54.929379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:54.929397 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:36:54.929509 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:54.929513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:54.929608 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:54.929740 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:36:54.930098 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.930810 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.932442 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.932993 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.933093 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:54.933551 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:36:54.934090 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:36:54.934203 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:54.934297 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 23:36:54.934363 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:36:54.943058 5066 raft_consensus.cc:1261] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392 [term 1 FOLLOWER]: Refusing update from remote peer 3c54206c5ec34327a6cf8e37f53de11b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:36:54.943056 5303 raft_consensus.cc:1261] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3 [term 1 FOLLOWER]: Refusing update from remote peer 3c54206c5ec34327a6cf8e37f53de11b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:36:54.944087 5432 consensus_queue.cc:1035] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2587d021327147b29fe6d84d679e1ca3" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 34885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:36:54.944451 5428 consensus_queue.cc:1035] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [LEADER]: Connected to new peer: Peer: permanent_uuid: "93262860f53b457aabd5a837aae4c392" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:36:54.957793 5441 mvcc.cc:205] Tried to move back new op lower bound from 7317114941196627968 to 7317114940988510208. Current Snapshot: MvccSnapshot[applied={T|T < 7317114941196627968}]
I20260810 23:36:54.970515 4633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:36:54.970796 4633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:36:54.974496 4633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b0c81ab46ca41569f978ce9fa2e35b4: Sending DeleteTablet for 3 replicas of tablet c31ed8be62a04d949bbc6d217a0fa106
I20260810 23:36:54.975648 5037 tablet_service.cc:1514] Processing DeleteTablet for tablet c31ed8be62a04d949bbc6d217a0fa106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:36:54 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:60990
I20260810 23:36:54.975819 4824 tablet_service.cc:1514] Processing DeleteTablet for tablet c31ed8be62a04d949bbc6d217a0fa106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:36:54 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:37552
I20260810 23:36:54.976568 5265 tablet_service.cc:1514] Processing DeleteTablet for tablet c31ed8be62a04d949bbc6d217a0fa106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:36:54 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:53018
I20260810 23:36:54.976975 5458 tablet_replica.cc:330] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392: stopping tablet replica
I20260810 23:36:54.977438 5458 raft_consensus.cc:2229] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:36:54.977772 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 4728
I20260810 23:36:54.977962 5458 raft_consensus.cc:2258] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:36:54.978744 5460 tablet_replica.cc:330] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3: stopping tablet replica
I20260810 23:36:54.979504 5460 raft_consensus.cc:2229] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:36:54.979979 5460 raft_consensus.cc:2258] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:36:54.980878 5459 tablet_replica.cc:330] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b: stopping tablet replica
I20260810 23:36:54.981402 5459 raft_consensus.cc:2229] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:36:54.981964 5458 ts_tablet_manager.cc:1905] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:36:54.982371 5459 raft_consensus.cc:2258] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:36:54.985370 5460 ts_tablet_manager.cc:1905] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:36:54.989598 5458 ts_tablet_manager.cc:1918] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:36:54.989846 5458 log.cc:1196] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/c31ed8be62a04d949bbc6d217a0fa106
I20260810 23:36:54.990473 5458 ts_tablet_manager.cc:1939] T c31ed8be62a04d949bbc6d217a0fa106 P 93262860f53b457aabd5a837aae4c392: Deleting consensus metadata
I20260810 23:36:54.992508 4619 catalog_manager.cc:4841] TS 93262860f53b457aabd5a837aae4c392 (127.9.48.194:43767): tablet c31ed8be62a04d949bbc6d217a0fa106 (table test-table [id=239acc13c1274de2b2b8f844cbcae7d3]) successfully deleted
I20260810 23:36:54.994289 5459 ts_tablet_manager.cc:1905] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:36:54.995038 5460 ts_tablet_manager.cc:1918] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:36:54.995283 5460 log.cc:1196] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/c31ed8be62a04d949bbc6d217a0fa106
I20260810 23:36:54.995954 5460 ts_tablet_manager.cc:1939] T c31ed8be62a04d949bbc6d217a0fa106 P 2587d021327147b29fe6d84d679e1ca3: Deleting consensus metadata
I20260810 23:36:54.997869 4617 catalog_manager.cc:4841] TS 2587d021327147b29fe6d84d679e1ca3 (127.9.48.195:34885): tablet c31ed8be62a04d949bbc6d217a0fa106 (table test-table [id=239acc13c1274de2b2b8f844cbcae7d3]) successfully deleted
I20260810 23:36:55.001830 5459 ts_tablet_manager.cc:1918] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:36:55.002020 5459 log.cc:1196] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/c31ed8be62a04d949bbc6d217a0fa106
I20260810 23:36:55.002506 5459 ts_tablet_manager.cc:1939] T c31ed8be62a04d949bbc6d217a0fa106 P 3c54206c5ec34327a6cf8e37f53de11b: Deleting consensus metadata
Aug 10 23:36:55 dist-test-slave-04bb krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:36:55.004139 4620 catalog_manager.cc:4841] TS 3c54206c5ec34327a6cf8e37f53de11b (127.9.48.193:32933): tablet c31ed8be62a04d949bbc6d217a0fa106 (table test-table [id=239acc13c1274de2b2b8f844cbcae7d3]) successfully deleted
I20260810 23:36:55.011583 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:54.981045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37572 (local address 127.9.48.193:32933)
0810 23:36:54.981223 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:54.981248 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:54.993061 (+ 11813us) server_negotiation.cc:407] Connection header received
0810 23:36:54.993289 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:54.993296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:54.993391 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:54.993463 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:54.995417 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.996761 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.998828 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:54.999582 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:54.999743 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:55.005318 (+ 5575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:55.005330 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:55.005345 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:55.005388 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:55.008578 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:55.009361 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:55.009369 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:55.009372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:55.009514 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:55.009961 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:55.009967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:55.009970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:55.010331 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:55.010642 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:55.010971 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 23:36:55.011223 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:36:55.012329 4911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:36:55.113814 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 4728
I20260810 23:36:55.142701 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 4937
Aug 10 23:36:55 dist-test-slave-04bb krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:36:55.159397 5405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:55.143582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.9.48.194:43767)
0810 23:36:55.143785 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:55.143790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:55.143811 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:36:55.143933 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:55.143937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:55.144021 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:55.144107 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:55.145516 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:55.146458 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:55.147812 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:55.148231 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:55.148350 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:55.152932 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:55.152948 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:55.152965 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:55.153010 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:55.156187 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:55.157164 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:55.157175 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:55.157179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:55.157325 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:55.157779 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:55.157783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:55.157785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:55.158211 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:55.158459 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:55.158799 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 23:36:55.158971 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:36:55.160173 5131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:36:55.251446 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 4937
I20260810 23:36:55.279414 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 5151
Aug 10 23:36:55 dist-test-slave-04bb krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:36:55.295181 5408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:55.280346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.9.48.195:34885)
0810 23:36:55.280576 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:55.280582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:55.280609 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:36:55.280806 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:55.280811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:55.280893 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:55.280988 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:55.282331 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:55.283095 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:55.284378 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:55.284860 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:55.284958 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:55.289710 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:55.289726 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:55.289743 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:36:55.289783 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:55.292532 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:55.293313 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:55.293321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:55.293323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:55.293460 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:36:55.293844 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:36:55.293851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:36:55.293854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:36:55.294226 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:55.294450 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:55.294744 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 23:36:55.294903 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:36:55.295872 5369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:36:55.385041 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 5151
I20260810 23:36:55.411604 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 4592
I20260810 23:36:55.412879 4686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:36:55.523389 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 4592
2026-08-10T23:36:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4781 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T23:36:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:36:55Z Disabled control of system clock
I20260810 23:36:55.628361 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:40709
--webserver_interface=127.9.48.254
--webserver_port=0
--builtin_ntp_servers=127.9.48.212:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:40709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:36:55.975040 5472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:55.975337 5472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:55.975420 5472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:55.982569 5472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:36:55.982678 5472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:55.982717 5472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:55.982779 5472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:36:55.982824 5472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:36:55.992749 5472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:40709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:40709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:55.994163 5472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:55.996007 5472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:56.010106 5485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:56.010040 5488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:56.010772 5483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:56.012046 5472 server_base.cc:1034] running on GCE node
I20260810 23:36:56.012988 5472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:56.014559 5472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:56.015769 5472 hybrid_clock.cc:648] HybridClock initialized: now 1786405016015677 us; error 82 us; skew 500 ppm
I20260810 23:36:56.016160 5472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:36:56.018682 5472 webserver.cc:460] Webserver started at http://127.9.48.254:39519/ using document root <none> and password file <none>
I20260810 23:36:56.019438 5472 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:56.019562 5472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:56.019923 5472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:36:56.023061 5472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "fa4a5edc6daa4bc697ae01d03fdfb529"
format_stamp: "Formatted at 2026-08-10 23:36:56 on dist-test-slave-04bb"
server_key: "495a7641cab68ed141f65066fc18a433"
server_key_iv: "a40519c382e26b0e78f8ee37ed3cbe63"
server_key_version: "encryptionkey@0"
I20260810 23:36:56.023936 5472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "fa4a5edc6daa4bc697ae01d03fdfb529"
format_stamp: "Formatted at 2026-08-10 23:36:56 on dist-test-slave-04bb"
server_key: "495a7641cab68ed141f65066fc18a433"
server_key_iv: "a40519c382e26b0e78f8ee37ed3cbe63"
server_key_version: "encryptionkey@0"
I20260810 23:36:56.029650 5472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:36:56.033370 5504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:56.034745 5472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:36:56.034950 5472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "fa4a5edc6daa4bc697ae01d03fdfb529"
format_stamp: "Formatted at 2026-08-10 23:36:56 on dist-test-slave-04bb"
server_key: "495a7641cab68ed141f65066fc18a433"
server_key_iv: "a40519c382e26b0e78f8ee37ed3cbe63"
server_key_version: "encryptionkey@0"
I20260810 23:36:56.035197 5472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:56.050233 5472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:56.051065 5472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:56.051386 5472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:56.071324 5472 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:40709
I20260810 23:36:56.071364 5588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:40709 every 8 connection(s)
I20260810 23:36:56.072793 5472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:36:56.075518 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 5472
I20260810 23:36:56.075736 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:36:56.076336 9411 external_mini_cluster.cc:1428] Setting key 63705c6be09ca4fb6bdc7a4cd6328e19
I20260810 23:36:56.077271 5591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:36:56.091501 5591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529: Bootstrap starting.
I20260810 23:36:56.091838 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:56.078367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43516 (local address 127.9.48.254:40709)
0810 23:36:56.079442 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:56.079453 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:56.079495 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:36:56.082052 (+ 2557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:56.082075 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:56.082666 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:56.083020 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:56.083708 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:56.085143 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:56.087203 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:56.087905 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:56.088043 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:56.089386 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:56.089394 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:56.089411 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:36:56.089957 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:56.090033 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:36:56.090054 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:56.090148 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:56.090398 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 23:36:56.090742 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":407,"threads_started":1}
I20260810 23:36:56.095297 5591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529: Neither blocks nor log segments found. Creating new log.
I20260810 23:36:56.096570 5591 log.cc:824] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529: Log is configured to *not* fsync() on all Append() calls
I20260810 23:36:56.100028 5591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529: No bootstrap required, opened a new log
I20260810 23:36:56.107251 5591 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4a5edc6daa4bc697ae01d03fdfb529" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40709 } }
I20260810 23:36:56.107637 5591 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:36:56.107784 5591 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4a5edc6daa4bc697ae01d03fdfb529, State: Initialized, Role: FOLLOWER
I20260810 23:36:56.108435 5591 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4a5edc6daa4bc697ae01d03fdfb529" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40709 } }
I20260810 23:36:56.108732 5591 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:36:56.108842 5591 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:36:56.108973 5591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:36:56.111032 5591 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4a5edc6daa4bc697ae01d03fdfb529" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40709 } }
I20260810 23:36:56.111492 5591 leader_election.cc:302] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa4a5edc6daa4bc697ae01d03fdfb529; no voters:
I20260810 23:36:56.112039 5591 leader_election.cc:288] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:36:56.112216 5597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:36:56.112963 5597 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [term 1 LEADER]: Becoming Leader. State: Replica: fa4a5edc6daa4bc697ae01d03fdfb529, State: Running, Role: LEADER
I20260810 23:36:56.113540 5597 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4a5edc6daa4bc697ae01d03fdfb529" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40709 } }
I20260810 23:36:56.114208 5591 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:36:56.116773 5600 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa4a5edc6daa4bc697ae01d03fdfb529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4a5edc6daa4bc697ae01d03fdfb529" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40709 } } }
I20260810 23:36:56.116830 5601 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa4a5edc6daa4bc697ae01d03fdfb529. Latest consensus state: current_term: 1 leader_uuid: "fa4a5edc6daa4bc697ae01d03fdfb529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa4a5edc6daa4bc697ae01d03fdfb529" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 40709 } } }
I20260810 23:36:56.117245 5600 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [sys.catalog]: This master's current role is: LEADER
I20260810 23:36:56.117316 5601 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529 [sys.catalog]: This master's current role is: LEADER
I20260810 23:36:56.119549 5608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:36:56.124917 5608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:36:56.134027 5608 catalog_manager.cc:1347] Generated new cluster ID: ec68ed2474144dec8b03bb3d4de7cead
I20260810 23:36:56.134171 5608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:36:56.152213 5608 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:36:56.153612 5608 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:36:56.161533 5608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529: Generated new TSK 0
I20260810 23:36:56.162396 5608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:36:56.173802 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40709
--builtin_ntp_servers=127.9.48.212:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:36:56.574620 5623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:56.574908 5623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:56.575065 5623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:56.583096 5623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:56.583232 5623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:56.583465 5623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:36:56.594355 5623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:56.595817 5623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:56.597950 5623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:56.612237 5635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:56.612149 5636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:56.612115 5638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:56.613298 5623 server_base.cc:1034] running on GCE node
I20260810 23:36:56.614037 5623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:56.615087 5623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:56.616348 5623 hybrid_clock.cc:648] HybridClock initialized: now 1786405016616301 us; error 43 us; skew 500 ppm
I20260810 23:36:56.616756 5623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:36:56.619309 5623 webserver.cc:460] Webserver started at http://127.9.48.193:42105/ using document root <none> and password file <none>
I20260810 23:36:56.620091 5623 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:56.620229 5623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:56.620693 5623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:36:56.623574 5623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342"
format_stamp: "Formatted at 2026-08-10 23:36:56 on dist-test-slave-04bb"
server_key: "7819f1d325c8373d8701864b92197bde"
server_key_iv: "a60c96368e1c3e538599a66bfed725c0"
server_key_version: "encryptionkey@0"
I20260810 23:36:56.624399 5623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342"
format_stamp: "Formatted at 2026-08-10 23:36:56 on dist-test-slave-04bb"
server_key: "7819f1d325c8373d8701864b92197bde"
server_key_iv: "a60c96368e1c3e538599a66bfed725c0"
server_key_version: "encryptionkey@0"
I20260810 23:36:56.629743 5623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:36:56.633392 5647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:56.634846 5623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:36:56.635046 5623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342"
format_stamp: "Formatted at 2026-08-10 23:36:56 on dist-test-slave-04bb"
server_key: "7819f1d325c8373d8701864b92197bde"
server_key_iv: "a60c96368e1c3e538599a66bfed725c0"
server_key_version: "encryptionkey@0"
I20260810 23:36:56.635293 5623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:56.647640 5623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:56.648478 5623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:56.648849 5623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:56.649858 5623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:36:56.651371 5623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:36:56.651499 5623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:56.651645 5623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:36:56.651736 5623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:56.686102 5623 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:37801
I20260810 23:36:56.686196 5816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:37801 every 8 connection(s)
I20260810 23:36:56.687515 5623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:36:56.692809 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 5623
I20260810 23:36:56.693004 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:36:56.693475 9411 external_mini_cluster.cc:1428] Setting key 5233dbf90fe21d17ad2bac61b83351f4
I20260810 23:36:56.696267 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40709
--builtin_ntp_servers=127.9.48.212:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:36:56.700661 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:56.690240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:41179 (local address 127.9.48.254:40709)
0810 23:36:56.690752 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:56.690757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:56.691829 (+ 1072us) server_negotiation.cc:407] Connection header received
0810 23:36:56.693909 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:56.693915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:56.694024 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:56.694159 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:56.695262 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:56.696003 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:56.697748 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:56.698160 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:56.698305 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:56.699748 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:56.699753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:56.699757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:36:56.699774 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:56.699815 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:36:56.699828 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:56.699932 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:56.700169 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 23:36:56.700268 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":272,"threads_started":1}
I20260810 23:36:56.701663 5829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:56.690723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40709 (local address 127.9.48.193:41179)
0810 23:36:56.691653 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:56.691699 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:56.693628 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:56.694398 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:56.694406 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:56.694493 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:36:56.695075 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:56.695093 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:56.696174 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:56.696185 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:56.697584 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:56.697607 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:56.698410 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:56.698418 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:56.698662 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:56.699406 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:36:56.699409 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:36:56.699450 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:36:56.699526 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:36:56.699934 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:36:56.699949 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:36:56.699959 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 23:36:56.700156 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 23:36:56.700564 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":293,"threads_started":1}
I20260810 23:36:56.703346 5818 heartbeater.cc:347] Connected to a master server at 127.9.48.254:40709
I20260810 23:36:56.703671 5818 heartbeater.cc:464] Registering TS with master...
I20260810 23:36:56.704586 5818 heartbeater.cc:511] Master 127.9.48.254:40709 requested a full tablet report, sending...
I20260810 23:36:56.707031 5537 ts_manager.cc:194] Registered new tserver with Master: 05dc62154e2e4cd6a8a0e7c51c7a6342 (127.9.48.193:37801)
I20260810 23:36:56.709230 5537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.48.193:41179
W20260810 23:36:57.050266 5830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:57.050616 5830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:57.050791 5830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:57.058192 5830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:57.058328 5830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:57.058501 5830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:36:57.067559 5830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:57.069026 5830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:57.070950 5830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:57.084220 5839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:57.084309 5838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:57.084863 5841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:57.085129 5830 server_base.cc:1034] running on GCE node
I20260810 23:36:57.085757 5830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:57.086771 5830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:57.087997 5830 hybrid_clock.cc:648] HybridClock initialized: now 1786405017087941 us; error 50 us; skew 500 ppm
I20260810 23:36:57.088328 5830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:36:57.090685 5830 webserver.cc:460] Webserver started at http://127.9.48.194:40807/ using document root <none> and password file <none>
I20260810 23:36:57.091387 5830 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:57.091480 5830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:57.091892 5830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:36:57.094746 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "58f75554543743bc8639329e1c01ffbf"
format_stamp: "Formatted at 2026-08-10 23:36:57 on dist-test-slave-04bb"
server_key: "fea93b5d1011205581269fc7505e2317"
server_key_iv: "defeac9d0034f9f7661a1e3eeb8c937e"
server_key_version: "encryptionkey@0"
I20260810 23:36:57.095477 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "58f75554543743bc8639329e1c01ffbf"
format_stamp: "Formatted at 2026-08-10 23:36:57 on dist-test-slave-04bb"
server_key: "fea93b5d1011205581269fc7505e2317"
server_key_iv: "defeac9d0034f9f7661a1e3eeb8c937e"
server_key_version: "encryptionkey@0"
I20260810 23:36:57.100471 5830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:36:57.103904 5849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:57.105266 5830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:36:57.105435 5830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "58f75554543743bc8639329e1c01ffbf"
format_stamp: "Formatted at 2026-08-10 23:36:57 on dist-test-slave-04bb"
server_key: "fea93b5d1011205581269fc7505e2317"
server_key_iv: "defeac9d0034f9f7661a1e3eeb8c937e"
server_key_version: "encryptionkey@0"
I20260810 23:36:57.105654 5830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:57.141204 5830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:57.142078 5830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:57.142385 5830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:57.143419 5830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:36:57.144951 5830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:36:57.145066 5830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:57.145195 5830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:36:57.145296 5830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:57.180039 5830 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:46689
I20260810 23:36:57.180132 6028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:46689 every 8 connection(s)
I20260810 23:36:57.181478 5830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:36:57.181854 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 5830
I20260810 23:36:57.182060 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:36:57.182726 9411 external_mini_cluster.cc:1428] Setting key d48311773a3b0a7fab0cb5ed7a74093d
I20260810 23:36:57.186283 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40709
--builtin_ntp_servers=127.9.48.212:44129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:36:57.195587 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.184689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:55493 (local address 127.9.48.254:40709)
0810 23:36:57.184898 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:57.184904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:57.186549 (+ 1645us) server_negotiation.cc:407] Connection header received
0810 23:36:57.188411 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:57.188420 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:57.188511 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:57.188649 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:57.189659 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.190819 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.192379 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.193129 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.193269 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:57.194648 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:57.194657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:57.194664 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:36:57.194690 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:57.194742 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:36:57.194764 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:57.194865 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:57.195076 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 23:36:57.195190 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:36:57.196447 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.185346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40709 (local address 127.9.48.194:55493)
0810 23:36:57.186388 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:57.186443 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:57.188134 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:57.188842 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:57.188850 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:57.188933 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:36:57.189492 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:57.189509 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.190984 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.190988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:57.192229 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:57.192242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.193354 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.193359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:57.193563 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:57.194328 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:36:57.194330 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:36:57.194369 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:36:57.194445 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:36:57.194871 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:36:57.194879 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:36:57.194885 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:36:57.195031 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 23:36:57.195332 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":346,"threads_started":1}
I20260810 23:36:57.197970 6029 heartbeater.cc:347] Connected to a master server at 127.9.48.254:40709
I20260810 23:36:57.198280 6029 heartbeater.cc:464] Registering TS with master...
I20260810 23:36:57.199038 6029 heartbeater.cc:511] Master 127.9.48.254:40709 requested a full tablet report, sending...
I20260810 23:36:57.200507 5537 ts_manager.cc:194] Registered new tserver with Master: 58f75554543743bc8639329e1c01ffbf (127.9.48.194:46689)
I20260810 23:36:57.201397 5537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.48.194:55493
W20260810 23:36:57.532575 6036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:36:57.532850 6036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:36:57.533034 6036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:36:57.540542 6036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:36:57.540635 6036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:36:57.540817 6036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:36:57.549840 6036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:40709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:36:57.551105 6036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:36:57.552985 6036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:36:57.567642 6044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:57.567664 6051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:36:57.567696 6047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:36:57.568768 6036 server_base.cc:1034] running on GCE node
I20260810 23:36:57.569450 6036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:36:57.570464 6036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:36:57.571708 6036 hybrid_clock.cc:648] HybridClock initialized: now 1786405017571654 us; error 46 us; skew 500 ppm
I20260810 23:36:57.572048 6036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:36:57.574599 6036 webserver.cc:460] Webserver started at http://127.9.48.195:44025/ using document root <none> and password file <none>
I20260810 23:36:57.575344 6036 fs_manager.cc:359] Metadata directory not provided
I20260810 23:36:57.575488 6036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:36:57.575848 6036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:36:57.578832 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "36d53ce50f694598b7a7cd73c57d10ce"
format_stamp: "Formatted at 2026-08-10 23:36:57 on dist-test-slave-04bb"
server_key: "4d8426b081050520a9cf42172f2a0207"
server_key_iv: "44ef35c01a89720a02009c8a04640406"
server_key_version: "encryptionkey@0"
I20260810 23:36:57.579661 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "36d53ce50f694598b7a7cd73c57d10ce"
format_stamp: "Formatted at 2026-08-10 23:36:57 on dist-test-slave-04bb"
server_key: "4d8426b081050520a9cf42172f2a0207"
server_key_iv: "44ef35c01a89720a02009c8a04640406"
server_key_version: "encryptionkey@0"
I20260810 23:36:57.584985 6036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:36:57.588481 6060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:57.589931 6036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:36:57.590096 6036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "36d53ce50f694598b7a7cd73c57d10ce"
format_stamp: "Formatted at 2026-08-10 23:36:57 on dist-test-slave-04bb"
server_key: "4d8426b081050520a9cf42172f2a0207"
server_key_iv: "44ef35c01a89720a02009c8a04640406"
server_key_version: "encryptionkey@0"
I20260810 23:36:57.590334 6036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:36:57.602111 6036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:36:57.602952 6036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:36:57.603267 6036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:36:57.604226 6036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:36:57.605839 6036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:36:57.605964 6036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:57.606099 6036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:36:57.606187 6036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:36:57.642321 6036 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:35601
I20260810 23:36:57.642411 6232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:35601 every 8 connection(s)
I20260810 23:36:57.643676 6036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:36:57.650502 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 6036
I20260810 23:36:57.650696 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:36:57.651206 9411 external_mini_cluster.cc:1428] Setting key 67ae0c9aab2f2f0a83e5683d0500282d
I20260810 23:36:57.657246 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.646361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:37645 (local address 127.9.48.254:40709)
0810 23:36:57.646564 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:57.646569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:57.648146 (+ 1577us) server_negotiation.cc:407] Connection header received
0810 23:36:57.650070 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:57.650078 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:57.650180 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:57.650332 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:57.651537 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.652558 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.654476 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.654871 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.655004 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:57.656366 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:57.656374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:57.656384 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:36:57.656401 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:57.656433 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:36:57.656445 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:57.656564 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:57.656803 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 23:36:57.656901 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:36:57.658219 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.646811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:40709 (local address 127.9.48.195:37645)
0810 23:36:57.647972 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:57.648021 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:57.649772 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:57.650526 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:57.650534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:57.650675 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:36:57.651362 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:57.651382 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.652732 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.652738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:57.654336 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:57.654349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.655097 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.655104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:57.655340 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:57.656046 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:36:57.656049 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:36:57.656094 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:36:57.656167 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:36:57.656566 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:36:57.656580 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:36:57.656590 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 23:36:57.656797 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 23:36:57.657174 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":322,"threads_started":1}
I20260810 23:36:57.659538 6234 heartbeater.cc:347] Connected to a master server at 127.9.48.254:40709
I20260810 23:36:57.659962 6234 heartbeater.cc:464] Registering TS with master...
I20260810 23:36:57.660823 6234 heartbeater.cc:511] Master 127.9.48.254:40709 requested a full tablet report, sending...
I20260810 23:36:57.662269 5537 ts_manager.cc:194] Registered new tserver with Master: 36d53ce50f694598b7a7cd73c57d10ce (127.9.48.195:35601)
I20260810 23:36:57.663066 5537 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.48.195:37645
I20260810 23:36:57.666455 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:36:57.676687 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.670592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43526 (local address 127.9.48.254:40709)
0810 23:36:57.670762 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:57.670766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:57.671264 (+ 498us) server_negotiation.cc:407] Connection header received
0810 23:36:57.671474 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:57.671478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:57.671550 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:57.671654 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:57.672173 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.672932 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.674312 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.674725 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.674858 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:57.675946 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:57.675951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:57.675954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:36:57.675971 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:57.676001 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:36:57.676014 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:57.676107 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:57.676273 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 23:36:57.676373 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:36:57.680406 5537 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:36:57.687191 5537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:36:57.712199 5818 heartbeater.cc:503] Master 127.9.48.254:40709 was elected leader, sending a full tablet report...
I20260810 23:36:57.713277 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.700025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:37801 (local address 127.0.0.1:54010)
0810 23:36:57.702804 (+ 2779us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:57.702822 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:57.702944 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:57.703940 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:57.703949 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:57.703985 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:36:57.704511 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:57.704546 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.706811 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.706816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:57.709830 (+ 3014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:57.709851 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.712199 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.712206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:57.712456 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:57.712514 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 23:36:57.712656 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 23:36:57.712881 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2610,"mutex_wait_us":126,"thread_start_us":1274,"threads_started":1}
I20260810 23:36:57.713414 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.699684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.9.48.194:46689)
0810 23:36:57.700961 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:57.700974 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:57.701964 (+ 990us) server_negotiation.cc:407] Connection header received
0810 23:36:57.702661 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:57.702674 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:57.702956 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:57.703137 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:36:57.704405 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.706520 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.710519 (+ 3999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.711933 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.712072 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:57.712325 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:57.712892 (+ 567us) server_negotiation.cc:299] Negotiation successful
0810 23:36:57.713072 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1052,"thread_start_us":891,"threads_started":1}
I20260810 23:36:57.714110 6257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.699641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:46689 (local address 127.0.0.1:58188)
0810 23:36:57.701776 (+ 2135us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:57.701827 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:57.702147 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:57.703538 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:57.703549 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:57.703592 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:36:57.704091 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:57.704109 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.706762 (+ 2653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.706771 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:57.710005 (+ 3234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:57.710017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.712461 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.712467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:57.712689 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:57.712719 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:36:57.713604 (+ 885us) client_negotiation.cc:240] Negotiation successful
0810 23:36:57.713742 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1816,"mutex_wait_us":423,"thread_start_us":2816,"threads_started":1}
I20260810 23:36:57.714321 6258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.700686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.9.48.193:37801)
0810 23:36:57.702454 (+ 1768us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:57.702464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:57.703092 (+ 628us) server_negotiation.cc:407] Connection header received
0810 23:36:57.703200 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:57.703209 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:57.703430 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:57.703628 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:36:57.704783 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.706613 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.710096 (+ 3483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.711610 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.711763 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:57.712063 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:57.713749 (+ 1686us) server_negotiation.cc:299] Negotiation successful
0810 23:36:57.713919 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1595,"thread_start_us":1465,"threads_started":1}
I20260810 23:36:57.717031 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.703608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:35601 (local address 127.0.0.1:37830)
0810 23:36:57.708249 (+ 4641us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:57.708276 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:57.708432 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:57.709446 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:57.709451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:57.709476 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:36:57.710045 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:57.710067 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.711736 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.711741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:57.713878 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:57.713892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.716163 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.716171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:57.716413 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:57.716447 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:36:57.716574 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 23:36:57.716697 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4448,"mutex_wait_us":4,"thread_start_us":4255,"threads_started":1}
I20260810 23:36:57.717175 6260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.703208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.9.48.195:35601)
0810 23:36:57.708983 (+ 5775us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:57.708990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:57.709017 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:36:57.709090 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:57.709095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:57.709257 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:57.709459 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:36:57.710261 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.711585 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.714150 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.715630 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.715785 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:57.716035 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:57.716652 (+ 617us) server_negotiation.cc:299] Negotiation successful
0810 23:36:57.716816 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5615,"thread_start_us":5461,"threads_started":1}
I20260810 23:36:57.718436 5723 tablet_service.cc:1467] Processing CreateTablet for tablet 3c3a8755f2304f53a3a966d451ace0ac (DEFAULT_TABLE table=test-table [id=a97523c6259e4e8d8dfa403de24e0dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:36:57.718358 5923 tablet_service.cc:1467] Processing CreateTablet for tablet 3c3a8755f2304f53a3a966d451ace0ac (DEFAULT_TABLE table=test-table [id=a97523c6259e4e8d8dfa403de24e0dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:36:57.719875 5723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c3a8755f2304f53a3a966d451ace0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:36:57.719890 5923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c3a8755f2304f53a3a966d451ace0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:36:57.720772 6135 tablet_service.cc:1467] Processing CreateTablet for tablet 3c3a8755f2304f53a3a966d451ace0ac (DEFAULT_TABLE table=test-table [id=a97523c6259e4e8d8dfa403de24e0dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:36:57.722177 6135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c3a8755f2304f53a3a966d451ace0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:36:57.731247 6262 tablet_bootstrap.cc:492] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342: Bootstrap starting.
I20260810 23:36:57.734396 6263 tablet_bootstrap.cc:492] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf: Bootstrap starting.
I20260810 23:36:57.734659 6262 tablet_bootstrap.cc:654] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342: Neither blocks nor log segments found. Creating new log.
I20260810 23:36:57.736011 6262 log.cc:824] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342: Log is configured to *not* fsync() on all Append() calls
I20260810 23:36:57.736212 6269 tablet_bootstrap.cc:492] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce: Bootstrap starting.
I20260810 23:36:57.738768 6263 tablet_bootstrap.cc:654] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf: Neither blocks nor log segments found. Creating new log.
I20260810 23:36:57.739640 6269 tablet_bootstrap.cc:654] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce: Neither blocks nor log segments found. Creating new log.
I20260810 23:36:57.740542 6263 log.cc:824] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf: Log is configured to *not* fsync() on all Append() calls
I20260810 23:36:57.740849 6269 log.cc:824] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce: Log is configured to *not* fsync() on all Append() calls
I20260810 23:36:57.743768 6269 tablet_bootstrap.cc:492] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce: No bootstrap required, opened a new log
I20260810 23:36:57.743920 6263 tablet_bootstrap.cc:492] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf: No bootstrap required, opened a new log
I20260810 23:36:57.743914 6262 tablet_bootstrap.cc:492] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342: No bootstrap required, opened a new log
I20260810 23:36:57.744093 6269 ts_tablet_manager.cc:1397] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260810 23:36:57.744184 6263 ts_tablet_manager.cc:1397] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 23:36:57.744189 6262 ts_tablet_manager.cc:1397] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260810 23:36:57.751420 6269 raft_consensus.cc:348] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75554543743bc8639329e1c01ffbf" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46689 } } peers { permanent_uuid: "36d53ce50f694598b7a7cd73c57d10ce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35601 } } peers { permanent_uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37801 } }
I20260810 23:36:57.751797 6269 raft_consensus.cc:374] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:36:57.751910 6269 raft_consensus.cc:729] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d53ce50f694598b7a7cd73c57d10ce, State: Initialized, Role: FOLLOWER
I20260810 23:36:57.752581 6269 consensus_queue.cc:260] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75554543743bc8639329e1c01ffbf" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46689 } } peers { permanent_uuid: "36d53ce50f694598b7a7cd73c57d10ce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35601 } } peers { permanent_uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37801 } }
I20260810 23:36:57.753798 6234 heartbeater.cc:503] Master 127.9.48.254:40709 was elected leader, sending a full tablet report...
I20260810 23:36:57.754504 6269 ts_tablet_manager.cc:1428] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260810 23:36:57.754886 6263 raft_consensus.cc:348] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75554543743bc8639329e1c01ffbf" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46689 } } peers { permanent_uuid: "36d53ce50f694598b7a7cd73c57d10ce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35601 } } peers { permanent_uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37801 } }
I20260810 23:36:57.754889 6262 raft_consensus.cc:348] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75554543743bc8639329e1c01ffbf" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46689 } } peers { permanent_uuid: "36d53ce50f694598b7a7cd73c57d10ce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35601 } } peers { permanent_uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37801 } }
I20260810 23:36:57.755308 6262 raft_consensus.cc:374] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:36:57.755308 6263 raft_consensus.cc:374] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:36:57.755415 6262 raft_consensus.cc:729] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05dc62154e2e4cd6a8a0e7c51c7a6342, State: Initialized, Role: FOLLOWER
I20260810 23:36:57.755419 6263 raft_consensus.cc:729] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f75554543743bc8639329e1c01ffbf, State: Initialized, Role: FOLLOWER
I20260810 23:36:57.756037 6262 consensus_queue.cc:260] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75554543743bc8639329e1c01ffbf" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46689 } } peers { permanent_uuid: "36d53ce50f694598b7a7cd73c57d10ce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35601 } } peers { permanent_uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37801 } }
I20260810 23:36:57.756031 6263 consensus_queue.cc:260] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75554543743bc8639329e1c01ffbf" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46689 } } peers { permanent_uuid: "36d53ce50f694598b7a7cd73c57d10ce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35601 } } peers { permanent_uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37801 } }
I20260810 23:36:57.758766 6029 heartbeater.cc:503] Master 127.9.48.254:40709 was elected leader, sending a full tablet report...
I20260810 23:36:57.759017 6262 ts_tablet_manager.cc:1428] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
I20260810 23:36:57.759364 6263 ts_tablet_manager.cc:1428] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260810 23:36:57.895457 6235 tablet.cc:2369] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:36:57.934056 6030 tablet.cc:2369] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:36:57.939900 6277 raft_consensus.cc:482] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260810 23:36:57.940198 5823 tablet.cc:2369] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:36:57.940352 6277 raft_consensus.cc:504] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75554543743bc8639329e1c01ffbf" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46689 } } peers { permanent_uuid: "36d53ce50f694598b7a7cd73c57d10ce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35601 } } peers { permanent_uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37801 } }
I20260810 23:36:57.942302 6277 leader_election.cc:288] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58f75554543743bc8639329e1c01ffbf (127.9.48.194:46689), 36d53ce50f694598b7a7cd73c57d10ce (127.9.48.195:35601)
I20260810 23:36:57.950186 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.942677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:56151 (local address 127.9.48.194:46689)
0810 23:36:57.942881 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:57.942887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:57.943547 (+ 660us) server_negotiation.cc:407] Connection header received
0810 23:36:57.943810 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:57.943821 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:57.943935 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:57.944003 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:36:57.944883 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.946071 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.948347 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.949128 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.949261 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:57.949320 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:57.949765 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 23:36:57.949917 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:36:57.950263 6260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.943929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:56163 (local address 127.9.48.195:35601)
0810 23:36:57.944158 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:57.944164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:57.944365 (+ 201us) server_negotiation.cc:407] Connection header received
0810 23:36:57.944628 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:57.944635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:57.944714 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:57.944823 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:36:57.945423 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.946303 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.948307 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.949334 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.949508 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:57.949579 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:57.949867 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 23:36:57.950015 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 23:36:57.950342 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.943564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:35601 (local address 127.9.48.193:56163)
0810 23:36:57.944253 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:57.944269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:57.944459 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:57.944837 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:57.944842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:57.944868 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:36:57.945249 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:57.945258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.946463 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.946472 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:57.948109 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:57.948125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.949515 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.949520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:57.949631 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:57.949695 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 23:36:57.949823 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 23:36:57.949945 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":38,"thread_start_us":315,"threads_started":1}
I20260810 23:36:57.950774 6279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.942672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:46689 (local address 127.9.48.193:56151)
0810 23:36:57.943433 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:57.943456 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:36:57.943600 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:36:57.944221 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:36:57.944225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:36:57.944260 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:36:57.944710 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:57.944726 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.946221 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.946227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:57.948139 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:36:57.948158 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:36:57.949333 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:36:57.949341 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:36:57.949539 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:57.949576 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:36:57.949685 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 23:36:57.949848 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":91,"spinlock_wait_cycles":48896,"thread_start_us":423,"threads_started":1}
I20260810 23:36:57.951462 6167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3a8755f2304f53a3a966d451ace0ac" candidate_uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d53ce50f694598b7a7cd73c57d10ce" is_pre_election: true
I20260810 23:36:57.951833 5957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3a8755f2304f53a3a966d451ace0ac" candidate_uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f75554543743bc8639329e1c01ffbf" is_pre_election: true
I20260810 23:36:57.952083 6167 raft_consensus.cc:2454] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05dc62154e2e4cd6a8a0e7c51c7a6342 in term 0.
I20260810 23:36:57.952376 5957 raft_consensus.cc:2454] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05dc62154e2e4cd6a8a0e7c51c7a6342 in term 0.
I20260810 23:36:57.952894 5650 leader_election.cc:302] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05dc62154e2e4cd6a8a0e7c51c7a6342, 36d53ce50f694598b7a7cd73c57d10ce; no voters:
I20260810 23:36:57.953292 6277 raft_consensus.cc:2781] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:36:57.953456 6277 raft_consensus.cc:482] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:36:57.953577 6277 raft_consensus.cc:3037] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:36:57.955971 6277 raft_consensus.cc:504] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75554543743bc8639329e1c01ffbf" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46689 } } peers { permanent_uuid: "36d53ce50f694598b7a7cd73c57d10ce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35601 } } peers { permanent_uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37801 } }
I20260810 23:36:57.956691 6277 leader_election.cc:288] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [CANDIDATE]: Term 1 election: Requested vote from peers 58f75554543743bc8639329e1c01ffbf (127.9.48.194:46689), 36d53ce50f694598b7a7cd73c57d10ce (127.9.48.195:35601)
I20260810 23:36:57.957147 5957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3a8755f2304f53a3a966d451ace0ac" candidate_uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f75554543743bc8639329e1c01ffbf"
I20260810 23:36:57.957350 5957 raft_consensus.cc:3037] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:36:57.957322 6167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c3a8755f2304f53a3a966d451ace0ac" candidate_uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36d53ce50f694598b7a7cd73c57d10ce"
I20260810 23:36:57.957530 6167 raft_consensus.cc:3037] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:36:57.959568 5957 raft_consensus.cc:2454] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05dc62154e2e4cd6a8a0e7c51c7a6342 in term 1.
I20260810 23:36:57.959925 5650 leader_election.cc:302] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05dc62154e2e4cd6a8a0e7c51c7a6342, 58f75554543743bc8639329e1c01ffbf; no voters:
I20260810 23:36:57.959952 6167 raft_consensus.cc:2454] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05dc62154e2e4cd6a8a0e7c51c7a6342 in term 1.
I20260810 23:36:57.960215 6277 raft_consensus.cc:2781] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:36:57.960767 6277 raft_consensus.cc:686] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [term 1 LEADER]: Becoming Leader. State: Replica: 05dc62154e2e4cd6a8a0e7c51c7a6342, State: Running, Role: LEADER
I20260810 23:36:57.961385 6277 consensus_queue.cc:237] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75554543743bc8639329e1c01ffbf" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46689 } } peers { permanent_uuid: "36d53ce50f694598b7a7cd73c57d10ce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35601 } } peers { permanent_uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37801 } }
I20260810 23:36:57.966251 5537 catalog_manager.cc:5495] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05dc62154e2e4cd6a8a0e7c51c7a6342 (127.9.48.193). New cstate: current_term: 1 leader_uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f75554543743bc8639329e1c01ffbf" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36d53ce50f694598b7a7cd73c57d10ce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05dc62154e2e4cd6a8a0e7c51c7a6342" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 37801 } health_report { overall_health: HEALTHY } } }
I20260810 23:36:58.005071 6258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:57.999552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.9.48.193:37801)
0810 23:36:57.999786 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:57.999811 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:57.999847 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:36:57.999964 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:57.999968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:58.000046 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:58.000210 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:36:58.000635 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:58.001326 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:58.002743 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:58.003174 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:58.003322 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:58.003904 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:36:58.004419 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:36:58.004600 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:58.004698 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 23:36:58.004776 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 23:36:58.014391 5957 raft_consensus.cc:1261] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf [term 1 FOLLOWER]: Refusing update from remote peer 05dc62154e2e4cd6a8a0e7c51c7a6342: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:36:58.014393 6167 raft_consensus.cc:1261] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce [term 1 FOLLOWER]: Refusing update from remote peer 05dc62154e2e4cd6a8a0e7c51c7a6342: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:36:58.015443 6281 consensus_queue.cc:1035] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f75554543743bc8639329e1c01ffbf" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 46689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:36:58.015805 6277 consensus_queue.cc:1035] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36d53ce50f694598b7a7cd73c57d10ce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 35601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:36:58.029412 6287 mvcc.cc:205] Tried to move back new op lower bound from 7317114953775276032 to 7317114953574129664. Current Snapshot: MvccSnapshot[applied={T|T < 7317114953775276032}]
I20260810 23:36:58.048247 5531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:36:58.048588 5531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:36:58.052763 5531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa4a5edc6daa4bc697ae01d03fdfb529: Sending DeleteTablet for 3 replicas of tablet 3c3a8755f2304f53a3a966d451ace0ac
I20260810 23:36:58.054158 5923 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c3a8755f2304f53a3a966d451ace0ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:36:58 UTC) from {username='slave'} at 127.0.0.1:58188
I20260810 23:36:58.054185 6135 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c3a8755f2304f53a3a966d451ace0ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:36:58 UTC) from {username='slave'} at 127.0.0.1:37830
I20260810 23:36:58.054847 5723 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c3a8755f2304f53a3a966d451ace0ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:36:58 UTC) from {username='slave'} at 127.0.0.1:54010
I20260810 23:36:58.056867 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 5623
I20260810 23:36:58.065758 6308 tablet_replica.cc:330] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342: stopping tablet replica
I20260810 23:36:58.066325 6308 raft_consensus.cc:2229] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:36:58.066741 6306 tablet_replica.cc:330] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce: stopping tablet replica
I20260810 23:36:58.066443 6304 tablet_replica.cc:330] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf: stopping tablet replica
I20260810 23:36:58.067147 6308 raft_consensus.cc:2258] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:36:58.067687 6304 raft_consensus.cc:2229] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:36:58.068310 6304 raft_consensus.cc:2258] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:36:58.068843 6306 raft_consensus.cc:2229] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:36:58.069432 6306 raft_consensus.cc:2258] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:36:58.071805 6306 ts_tablet_manager.cc:1905] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:36:58.073364 6304 ts_tablet_manager.cc:1905] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:36:58.079666 6306 ts_tablet_manager.cc:1918] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:36:58.080210 6258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:58.058420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54032 (local address 127.9.48.193:37801)
0810 23:36:58.058609 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:58.058615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:58.071534 (+ 12919us) server_negotiation.cc:407] Connection header received
0810 23:36:58.071856 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:58.071862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:58.071973 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:58.072058 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:58.072850 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:58.073970 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:58.075986 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:58.076728 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:58.076840 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:58.078419 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:58.078436 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:58.078455 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:36:58.079302 (+ 847us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:58.079364 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:36:58.079392 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:58.079472 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:58.079753 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 23:36:58.079872 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:36:58.079864 6306 log.cc:1196] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/3c3a8755f2304f53a3a966d451ace0ac
I20260810 23:36:58.080803 6308 ts_tablet_manager.cc:1905] T 3c3a8755f2304f53a3a966d451ace0ac P 05dc62154e2e4cd6a8a0e7c51c7a6342: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:36:58.081028 6306 ts_tablet_manager.cc:1939] T 3c3a8755f2304f53a3a966d451ace0ac P 36d53ce50f694598b7a7cd73c57d10ce: Deleting consensus metadata
I20260810 23:36:58.081595 5812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:36:58.083490 5508 catalog_manager.cc:4841] TS 36d53ce50f694598b7a7cd73c57d10ce (127.9.48.195:35601): tablet 3c3a8755f2304f53a3a966d451ace0ac (table test-table [id=a97523c6259e4e8d8dfa403de24e0dcb]) successfully deleted
I20260810 23:36:58.087881 6304 ts_tablet_manager.cc:1918] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:36:58.088052 6304 log.cc:1196] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/3c3a8755f2304f53a3a966d451ace0ac
I20260810 23:36:58.088651 6304 ts_tablet_manager.cc:1939] T 3c3a8755f2304f53a3a966d451ace0ac P 58f75554543743bc8639329e1c01ffbf: Deleting consensus metadata
I20260810 23:36:58.090536 5508 catalog_manager.cc:4841] TS 58f75554543743bc8639329e1c01ffbf (127.9.48.194:46689): tablet 3c3a8755f2304f53a3a966d451ace0ac (table test-table [id=a97523c6259e4e8d8dfa403de24e0dcb]) successfully deleted
I20260810 23:36:58.187754 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 5623
W20260810 23:36:58.218815 5509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.48.193:37801 (error 108)
I20260810 23:36:58.219286 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 5830
I20260810 23:36:58.221479 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:58.220149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:37801 (local address 127.0.0.1:54040)
0810 23:36:58.221073 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:58.221267 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":689,"threads_started":1}
W20260810 23:36:58.221730 5509 catalog_manager.cc:4584] TS 05dc62154e2e4cd6a8a0e7c51c7a6342 (127.9.48.193:37801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c3a8755f2304f53a3a966d451ace0ac: Network error: Client connection negotiation failed: client connection to 127.9.48.193:37801: connect: Connection refused (error 111)
I20260810 23:36:58.227202 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:58.220407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.9.48.194:46689)
0810 23:36:58.220773 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:58.220783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:58.220813 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:36:58.220897 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:58.220903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:58.221033 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:58.221154 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:58.221736 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:58.222560 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:58.223861 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:58.224277 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:58.224417 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:58.225776 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:58.225786 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:58.225801 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:36:58.226400 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:58.226450 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:36:58.226471 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:58.226611 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:58.226750 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 23:36:58.226864 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260810 23:36:58.227936 6024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:36:58.275507 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:58.275023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:37801 (local address 127.0.0.1:54044)
0810 23:36:58.275254 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:58.275352 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 23:36:58.276724 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:58.276261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:37801 (local address 127.0.0.1:54054)
0810 23:36:58.276456 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:58.276572 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:36:58.318965 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 5830
I20260810 23:36:58.337381 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:58.336871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:37801 (local address 127.0.0.1:54064)
0810 23:36:58.337099 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:58.337203 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 23:36:58.338624 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:58.338202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:37801 (local address 127.0.0.1:54070)
0810 23:36:58.338419 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:58.338487 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260810 23:36:58.346391 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 6036
I20260810 23:36:58.354050 6260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:58.347180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.9.48.195:35601)
0810 23:36:58.347399 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 23:36:58.347405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:36:58.347426 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:36:58.347616 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:36:58.347621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:36:58.347728 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:36:58.347865 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:36:58.348367 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:58.349221 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:58.350676 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:36:58.351110 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:36:58.351263 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:36:58.352452 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:36:58.352463 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:36:58.352479 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:36:58.353187 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:36:58.353247 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:36:58.353270 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:36:58.353411 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 23:36:58.353572 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 23:36:58.353700 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 23:36:58.354722 6227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:36:58.419615 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:58.419063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:37801 (local address 127.0.0.1:54072)
0810 23:36:58.419308 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:58.419424 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 23:36:58.421065 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:36:58.420438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:37801 (local address 127.0.0.1:54078)
0810 23:36:58.420786 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 23:36:58.420884 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237}
I20260810 23:36:58.444492 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 6036
I20260810 23:36:58.469136 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 5472
I20260810 23:36:58.470350 5582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:36:58.582706 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 5472
2026-08-10T23:36:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3053 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:36:58 dist-test-slave-04bb krb5kdc[6325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:36:58 dist-test-slave-04bb krb5kdc[6325](info): set up 2 sockets
Aug 10 23:36:58 dist-test-slave-04bb krb5kdc[6325](info): commencing operation
krb5kdc: starting...
W20260810 23:36:59.969580 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.334s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:00 dist-test-slave-04bb krb5kdc[6325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:37:00.303763 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:39893
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:39893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:00.653822 6348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:00.654104 6348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:00.654277 6348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:00.662009 6348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:00.662137 6348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:00.662232 6348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:00.662336 6348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:00.662420 6348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:00.672367 6348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:39893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:39893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:00.673827 6348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:00.675161 6348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:00.687302 6355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:00.687247 6356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:00.687279 6358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:00.688330 6348 server_base.cc:1034] running on GCE node
I20260810 23:37:00.689138 6348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:00.690551 6348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:00.691781 6348 hybrid_clock.cc:648] HybridClock initialized: now 1786405020691723 us; error 63 us; skew 500 ppm
I20260810 23:37:00.692128 6348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:00 dist-test-slave-04bb krb5kdc[6325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405020, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:00.695576 6348 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:37:00.697201 6348 webserver.cc:460] Webserver started at http://127.9.48.254:43827/ using document root <none> and password file <none>
I20260810 23:37:00.697917 6348 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:00.698071 6348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:00.698393 6348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:00.701464 6348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "543b227734304239a14c45c246ab8cd5"
format_stamp: "Formatted at 2026-08-10 23:37:00 on dist-test-slave-04bb"
server_key: "7cd71e8193f80811af325d8b289821ed"
server_key_iv: "cdd4691477460842525ffb33a67353aa"
server_key_version: "encryptionkey@0"
I20260810 23:37:00.702230 6348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "543b227734304239a14c45c246ab8cd5"
format_stamp: "Formatted at 2026-08-10 23:37:00 on dist-test-slave-04bb"
server_key: "7cd71e8193f80811af325d8b289821ed"
server_key_iv: "cdd4691477460842525ffb33a67353aa"
server_key_version: "encryptionkey@0"
I20260810 23:37:00.707465 6348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 23:37:00.711012 6366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:00.712289 6348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:37:00.712482 6348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "543b227734304239a14c45c246ab8cd5"
format_stamp: "Formatted at 2026-08-10 23:37:00 on dist-test-slave-04bb"
server_key: "7cd71e8193f80811af325d8b289821ed"
server_key_iv: "cdd4691477460842525ffb33a67353aa"
server_key_version: "encryptionkey@0"
I20260810 23:37:00.712769 6348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:00.726261 6348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:00.730773 6348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:00.731068 6348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:00.750479 6348 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:39893
I20260810 23:37:00.750512 6444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:39893 every 8 connection(s)
I20260810 23:37:00.751861 6348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:00.755295 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 6348
I20260810 23:37:00.755501 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:00.756033 6445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:00.756139 9411 external_mini_cluster.cc:1428] Setting key 56fd34abb9d2223b851877a102b20bc7
I20260810 23:37:00.771034 6445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5: Bootstrap starting.
Aug 10 23:37:00 dist-test-slave-04bb krb5kdc[6325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:37:00.775327 6445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:00.776834 6445 log.cc:824] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:00.780737 6445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5: No bootstrap required, opened a new log
I20260810 23:37:00.783213 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:00.758056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.9.48.254:39893)
0810 23:37:00.758839 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:00.758857 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:00.758914 (+ 57us) server_negotiation.cc:407] Connection header received
0810 23:37:00.761337 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:00.761366 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:00.761896 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:00.762341 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:00.763695 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:00.765771 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:00.767387 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:00.768549 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:00.768773 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:00.773433 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:00.773444 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:00.773465 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:00.773517 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:00.777843 (+ 4326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:00.778971 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:00.778980 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:00.778988 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:00.779277 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:00.779731 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:00.779738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:00.779740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:00.780451 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:00.780804 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:00.781346 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 23:37:00.781815 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":338,"threads_started":1}
I20260810 23:37:00.790259 6445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543b227734304239a14c45c246ab8cd5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39893 } }
I20260810 23:37:00.790596 6445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:00.790683 6445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 543b227734304239a14c45c246ab8cd5, State: Initialized, Role: FOLLOWER
I20260810 23:37:00.791152 6445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543b227734304239a14c45c246ab8cd5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39893 } }
I20260810 23:37:00.791360 6445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:00.791456 6445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:00.791579 6445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:00.793759 6445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543b227734304239a14c45c246ab8cd5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39893 } }
I20260810 23:37:00.794253 6445 leader_election.cc:302] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 543b227734304239a14c45c246ab8cd5; no voters:
I20260810 23:37:00.794772 6445 leader_election.cc:288] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:00.794931 6453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:00.796216 6453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [term 1 LEADER]: Becoming Leader. State: Replica: 543b227734304239a14c45c246ab8cd5, State: Running, Role: LEADER
I20260810 23:37:00.796821 6453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543b227734304239a14c45c246ab8cd5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39893 } }
I20260810 23:37:00.797148 6445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:00.799880 6455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "543b227734304239a14c45c246ab8cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543b227734304239a14c45c246ab8cd5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39893 } } }
I20260810 23:37:00.799858 6457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 543b227734304239a14c45c246ab8cd5. Latest consensus state: current_term: 1 leader_uuid: "543b227734304239a14c45c246ab8cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543b227734304239a14c45c246ab8cd5" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39893 } } }
I20260810 23:37:00.800360 6455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:00.800371 6457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:00.801630 6472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:00.808609 6472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:00.819734 6472 catalog_manager.cc:1347] Generated new cluster ID: d6075dd07609468bb232d8cc261c49f2
I20260810 23:37:00.819869 6472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:00.842597 6472 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:00.844472 6472 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:00.855285 6472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5: Generated new TSK 0
I20260810 23:37:00.856184 6472 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:37:00.944293 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39893
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:01.312848 6491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:01.313208 6491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:01.313388 6491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:01.321638 6491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:01.321796 6491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:01.321972 6491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:37:01.331890 6491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:01.333341 6491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:01.334802 6491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:01.347898 6498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:01.347976 6499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:01.348007 6501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:01.348841 6491 server_base.cc:1034] running on GCE node
I20260810 23:37:01.349457 6491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:01.350427 6491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:01.351666 6491 hybrid_clock.cc:648] HybridClock initialized: now 1786405021351635 us; error 57 us; skew 500 ppm
I20260810 23:37:01.351995 6491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:01 dist-test-slave-04bb krb5kdc[6325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405021, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:01.355362 6491 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:37:01.356886 6491 webserver.cc:460] Webserver started at http://127.9.48.193:36821/ using document root <none> and password file <none>
I20260810 23:37:01.357585 6491 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:01.357733 6491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:01.358062 6491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:01.361132 6491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "b261a06a0b484ee091cb0c8424a6bdbe"
format_stamp: "Formatted at 2026-08-10 23:37:01 on dist-test-slave-04bb"
server_key: "9ab6396e74d6d82af8f0190d6acedadd"
server_key_iv: "d7949b4d44b7cdd81abcc08e9c440a1b"
server_key_version: "encryptionkey@0"
I20260810 23:37:01.361934 6491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "b261a06a0b484ee091cb0c8424a6bdbe"
format_stamp: "Formatted at 2026-08-10 23:37:01 on dist-test-slave-04bb"
server_key: "9ab6396e74d6d82af8f0190d6acedadd"
server_key_iv: "d7949b4d44b7cdd81abcc08e9c440a1b"
server_key_version: "encryptionkey@0"
I20260810 23:37:01.367063 6491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:37:01.370615 6515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:01.372062 6491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:37:01.372236 6491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "b261a06a0b484ee091cb0c8424a6bdbe"
format_stamp: "Formatted at 2026-08-10 23:37:01 on dist-test-slave-04bb"
server_key: "9ab6396e74d6d82af8f0190d6acedadd"
server_key_iv: "d7949b4d44b7cdd81abcc08e9c440a1b"
server_key_version: "encryptionkey@0"
I20260810 23:37:01.372502 6491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:01.390249 6491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:01.394685 6491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:01.394987 6491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:01.395954 6491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:37:01.397473 6491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:37:01.397547 6491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:01.397672 6491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:37:01.397735 6491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:01.432210 6491 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:33181
I20260810 23:37:01.432262 6696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:33181 every 8 connection(s)
I20260810 23:37:01.433563 6491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:37:01.443099 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 6491
I20260810 23:37:01.443295 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:37:01.443856 9411 external_mini_cluster.cc:1428] Setting key b09c13445efcf200d2da332740e4f0f7
Aug 10 23:37:01 dist-test-slave-04bb krb5kdc[6325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405021, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:37:01.454591 6702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:01.436272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:40273 (local address 127.9.48.254:39893)
0810 23:37:01.436767 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:01.436772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:01.437690 (+ 918us) server_negotiation.cc:407] Connection header received
0810 23:37:01.439545 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:01.439551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:01.439652 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:01.439781 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:01.441346 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:01.442089 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:01.444326 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:01.444835 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:01.444949 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:01.448772 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:01.448777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:01.448780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:01.448812 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:01.451460 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:01.452188 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:01.452200 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:01.452203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:01.452326 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:01.452830 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:01.452837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:01.452845 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:01.453107 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:01.453268 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:01.453976 (+ 708us) server_negotiation.cc:299] Negotiation successful
0810 23:37:01.454210 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":278,"threads_started":1}
I20260810 23:37:01.455610 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:01.436847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39893 (local address 127.9.48.193:40273)
0810 23:37:01.437528 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:01.437573 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:01.439261 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:01.440035 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:01.440043 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:01.440592 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:01.441168 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:01.441186 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:01.442257 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:01.442267 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:01.444114 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:01.444130 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:01.445075 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:01.445081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:01.445280 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:01.446041 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:37:01.446069 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:01.448495 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:01.451604 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:01.451613 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:01.451629 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:01.452055 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:01.452441 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:01.452446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:01.452450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:01.452707 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:01.453248 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:01.453255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:01.453598 (+ 343us) client_negotiation.cc:769] Sending connection context
0810 23:37:01.453860 (+ 262us) client_negotiation.cc:240] Negotiation successful
0810 23:37:01.454403 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":265,"threads_started":1}
I20260810 23:37:01.457187 6699 heartbeater.cc:347] Connected to a master server at 127.9.48.254:39893
I20260810 23:37:01.457526 6699 heartbeater.cc:464] Registering TS with master...
I20260810 23:37:01.458432 6699 heartbeater.cc:511] Master 127.9.48.254:39893 requested a full tablet report, sending...
I20260810 23:37:01.461177 6393 ts_manager.cc:194] Registered new tserver with Master: b261a06a0b484ee091cb0c8424a6bdbe (127.9.48.193:33181)
I20260810 23:37:01.462707 6393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:40273
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:37:01.516036 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39893
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:01.866639 6711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:01.866921 6711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:01.867017 6711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:01.874367 6711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:01.874476 6711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:01.874589 6711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:37:01.885042 6711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:01.886409 6711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:01.887782 6711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:01.901353 6720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:01.901396 6717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:01.901518 6723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:01.902441 6711 server_base.cc:1034] running on GCE node
I20260810 23:37:01.903105 6711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:01.904110 6711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:01.905334 6711 hybrid_clock.cc:648] HybridClock initialized: now 1786405021905293 us; error 30 us; skew 500 ppm
I20260810 23:37:01.905628 6711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:01 dist-test-slave-04bb krb5kdc[6325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405021, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:01.909240 6711 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:37:01.910872 6711 webserver.cc:460] Webserver started at http://127.9.48.194:36337/ using document root <none> and password file <none>
I20260810 23:37:01.911607 6711 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:01.911765 6711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:01.912128 6711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:01.915397 6711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "2477767319ce4919a75f101d8746a59b"
format_stamp: "Formatted at 2026-08-10 23:37:01 on dist-test-slave-04bb"
server_key: "b0129e3977f5b89e5f054be8ddb9767e"
server_key_iv: "b47392f5ff51dd3b1b47a002e7e58248"
server_key_version: "encryptionkey@0"
I20260810 23:37:01.916307 6711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "2477767319ce4919a75f101d8746a59b"
format_stamp: "Formatted at 2026-08-10 23:37:01 on dist-test-slave-04bb"
server_key: "b0129e3977f5b89e5f054be8ddb9767e"
server_key_iv: "b47392f5ff51dd3b1b47a002e7e58248"
server_key_version: "encryptionkey@0"
I20260810 23:37:01.921919 6711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 23:37:01.925594 6733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:01.926997 6711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:37:01.927183 6711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "2477767319ce4919a75f101d8746a59b"
format_stamp: "Formatted at 2026-08-10 23:37:01 on dist-test-slave-04bb"
server_key: "b0129e3977f5b89e5f054be8ddb9767e"
server_key_iv: "b47392f5ff51dd3b1b47a002e7e58248"
server_key_version: "encryptionkey@0"
I20260810 23:37:01.927405 6711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:01.946156 6711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:01.950850 6711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:01.951164 6711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:01.952178 6711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:37:01.953739 6711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:37:01.953823 6711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:01.953919 6711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:37:01.953969 6711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:01.987528 6711 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:42193
I20260810 23:37:01.987574 6901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:42193 every 8 connection(s)
I20260810 23:37:01.989145 6711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:37:01.998346 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 6711
I20260810 23:37:01.998555 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:37:01.999125 9411 external_mini_cluster.cc:1428] Setting key 9a38b4135ddf92b4752f61c2f7935c54
Aug 10 23:37:01 dist-test-slave-04bb krb5kdc[6325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405021, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:37:02.012574 6905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:01.991853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:38287 (local address 127.9.48.254:39893)
0810 23:37:01.992313 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:01.992318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:01.993445 (+ 1127us) server_negotiation.cc:407] Connection header received
0810 23:37:01.995332 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:01.995337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:01.995439 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:01.995560 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:01.997241 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:01.998084 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.000255 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.000790 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.000945 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.005258 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:02.005267 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:02.005276 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:02.005324 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:02.009146 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:02.009917 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:02.009925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:02.009931 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:02.010093 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:02.010522 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:02.010530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:02.010536 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:02.010942 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:02.011098 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:02.011807 (+ 709us) server_negotiation.cc:299] Negotiation successful
0810 23:37:02.012061 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":221,"threads_started":1}
I20260810 23:37:02.013260 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:01.992446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39893 (local address 127.9.48.194:38287)
0810 23:37:01.993263 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:01.993312 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:01.995052 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:01.995809 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:01.995817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:01.996310 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:01.996947 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:01.996966 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:01.998288 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:01.998294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:02.000093 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:02.000106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.001053 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.001061 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:02.001289 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.002052 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:37:02.002086 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:02.005018 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:02.009315 (+ 4297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:02.009322 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:02.009335 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:02.009777 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:02.010206 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:02.010210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:02.010213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:02.010398 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:02.011082 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:02.011088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:02.011407 (+ 319us) client_negotiation.cc:769] Sending connection context
0810 23:37:02.011647 (+ 240us) client_negotiation.cc:240] Negotiation successful
0810 23:37:02.012029 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":301,"threads_started":1}
I20260810 23:37:02.014820 6902 heartbeater.cc:347] Connected to a master server at 127.9.48.254:39893
I20260810 23:37:02.015131 6902 heartbeater.cc:464] Registering TS with master...
I20260810 23:37:02.015856 6902 heartbeater.cc:511] Master 127.9.48.254:39893 requested a full tablet report, sending...
I20260810 23:37:02.017472 6393 ts_manager.cc:194] Registered new tserver with Master: 2477767319ce4919a75f101d8746a59b (127.9.48.194:42193)
I20260810 23:37:02.018273 6393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:38287
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:37:02.071292 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39893
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:02.423478 6914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:02.423878 6914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:02.424059 6914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:02.431198 6914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:02.431349 6914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:02.431571 6914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:37:02.441769 6914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:39893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:02.443054 6914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:02.444468 6914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:02.458106 6922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:02.458073 6921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:02.458194 6924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:02.458810 6914 server_base.cc:1034] running on GCE node
I20260810 23:37:02.459404 6914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:02.460392 6914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:02.461592 6914 hybrid_clock.cc:648] HybridClock initialized: now 1786405022461581 us; error 64 us; skew 500 ppm
I20260810 23:37:02.461921 6914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:02 dist-test-slave-04bb krb5kdc[6325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405022, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:02.465055 6914 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:37:02.465734 6699 heartbeater.cc:503] Master 127.9.48.254:39893 was elected leader, sending a full tablet report...
I20260810 23:37:02.466821 6914 webserver.cc:460] Webserver started at http://127.9.48.195:40753/ using document root <none> and password file <none>
I20260810 23:37:02.467579 6914 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:02.467757 6914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:02.468099 6914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:02.471145 6914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "3e20c7c22df24fdf9b89f868c4822090"
format_stamp: "Formatted at 2026-08-10 23:37:02 on dist-test-slave-04bb"
server_key: "4ed9cde74ec8788e46152be577c7e0ad"
server_key_iv: "ead7237735360ab22063efcd1eb5c35f"
server_key_version: "encryptionkey@0"
I20260810 23:37:02.471894 6914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "3e20c7c22df24fdf9b89f868c4822090"
format_stamp: "Formatted at 2026-08-10 23:37:02 on dist-test-slave-04bb"
server_key: "4ed9cde74ec8788e46152be577c7e0ad"
server_key_iv: "ead7237735360ab22063efcd1eb5c35f"
server_key_version: "encryptionkey@0"
I20260810 23:37:02.477403 6914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:37:02.480924 6933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:02.482260 6914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:37:02.482443 6914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "3e20c7c22df24fdf9b89f868c4822090"
format_stamp: "Formatted at 2026-08-10 23:37:02 on dist-test-slave-04bb"
server_key: "4ed9cde74ec8788e46152be577c7e0ad"
server_key_iv: "ead7237735360ab22063efcd1eb5c35f"
server_key_version: "encryptionkey@0"
I20260810 23:37:02.482645 6914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:02.499107 6914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:02.503355 6914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:02.503643 6914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:02.504678 6914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:37:02.506112 6914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:37:02.506220 6914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:02.506351 6914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:37:02.506399 6914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:02.539027 6914 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:40873
I20260810 23:37:02.539065 7109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:40873 every 8 connection(s)
I20260810 23:37:02.540311 6914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:37:02.545899 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 6914
I20260810 23:37:02.546083 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:37:02.546694 9411 external_mini_cluster.cc:1428] Setting key 64f3e7cd64e252a46c3f01cf5dedca87
Aug 10 23:37:02 dist-test-slave-04bb krb5kdc[6325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405022, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:37:02.560607 7118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.542914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:44809 (local address 127.9.48.254:39893)
0810 23:37:02.543386 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:02.543391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:02.544459 (+ 1068us) server_negotiation.cc:407] Connection header received
0810 23:37:02.546441 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:02.546448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:02.546549 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:02.546681 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:02.548508 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.549486 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.550967 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.551351 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.551445 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.554808 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:02.554813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:02.554816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:02.554849 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:02.557466 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:02.558199 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:02.558203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:02.558205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:02.558366 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:02.558817 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:02.558823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:02.558826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:02.559089 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:02.559276 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:02.559952 (+ 676us) server_negotiation.cc:299] Negotiation successful
0810 23:37:02.560177 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":231,"threads_started":1}
I20260810 23:37:02.561583 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.543495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:39893 (local address 127.9.48.195:44809)
0810 23:37:02.544259 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:02.544316 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:02.546154 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:02.546941 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:02.546949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:02.547614 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:02.548313 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:02.548339 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.549639 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.549643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:02.550840 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:02.550852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.551538 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.551543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:02.551738 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.552423 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:37:02.552450 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:02.554624 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:02.557618 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:02.557625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:02.557636 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:02.558079 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:02.558489 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:02.558495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:02.558499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:02.558694 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:02.559223 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:02.559232 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:02.559566 (+ 334us) client_negotiation.cc:769] Sending connection context
0810 23:37:02.559858 (+ 292us) client_negotiation.cc:240] Negotiation successful
0810 23:37:02.560366 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":264,"threads_started":1}
I20260810 23:37:02.562903 7116 heartbeater.cc:347] Connected to a master server at 127.9.48.254:39893
I20260810 23:37:02.563228 7116 heartbeater.cc:464] Registering TS with master...
I20260810 23:37:02.563958 7116 heartbeater.cc:511] Master 127.9.48.254:39893 requested a full tablet report, sending...
I20260810 23:37:02.565528 6393 ts_manager.cc:194] Registered new tserver with Master: 3e20c7c22df24fdf9b89f868c4822090 (127.9.48.195:40873)
I20260810 23:37:02.566394 6393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:44809
I20260810 23:37:02.574707 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:37:02.593281 7118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.579298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.9.48.254:39893)
0810 23:37:02.579564 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:02.579573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:02.579901 (+ 328us) server_negotiation.cc:407] Connection header received
0810 23:37:02.580033 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:02.580037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:02.580111 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:02.580205 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:02.581788 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.582592 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.583949 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.584392 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.584506 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.587432 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:02.587437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:02.587441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:02.587473 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:02.590371 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:02.591239 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:02.591242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:02.591245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:02.591382 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:02.591784 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:02.591787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:02.591789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:02.592078 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:02.592303 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:02.592767 (+ 464us) server_negotiation.cc:299] Negotiation successful
0810 23:37:02.592906 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 23:37:02.597270 6393 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:37:02.605181 6393 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:37:02.629869 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.619848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.9.48.195:40873)
0810 23:37:02.621697 (+ 1849us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:02.621706 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:02.621742 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:37:02.621848 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:02.621856 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:02.622101 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:02.623005 (+ 904us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:02.623285 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.624753 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.627459 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.628537 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.628979 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.629204 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:02.629330 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 23:37:02.629497 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1641,"thread_start_us":1450,"threads_started":1}
I20260810 23:37:02.629987 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.619485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:40873 (local address 127.0.0.1:39046)
0810 23:37:02.620032 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:02.620057 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:02.620185 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:02.622349 (+ 2164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:02.622354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:02.622381 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:02.622791 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:02.622804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.624936 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.624942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:02.627278 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:02.627295 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.628796 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.628802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:02.629133 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.629172 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 23:37:02.629272 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 23:37:02.629459 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":49,"thread_start_us":270,"threads_started":1}
I20260810 23:37:02.630884 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.618329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.9.48.193:33181)
0810 23:37:02.619398 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:02.619408 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:02.619443 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:37:02.619660 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:02.619672 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:02.619967 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:02.621732 (+ 1765us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:02.622002 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.623811 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.626879 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.628405 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.628594 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.628851 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:02.630445 (+ 1594us) server_negotiation.cc:299] Negotiation successful
0810 23:37:02.630618 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":633,"threads_started":1}
I20260810 23:37:02.631544 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.618015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:33181 (local address 127.0.0.1:54728)
0810 23:37:02.619101 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:02.619166 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:02.619397 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:02.620249 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:02.620258 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:02.620304 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:02.620902 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:02.620921 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.623991 (+ 3070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.624000 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:02.626691 (+ 2691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:02.626708 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.628673 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.628683 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:02.629007 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.629063 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 23:37:02.629207 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 23:37:02.631225 (+ 2018us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":25,"thread_start_us":535,"threads_started":1}
I20260810 23:37:02.633428 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.620112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:42193 (local address 127.0.0.1:41386)
0810 23:37:02.620857 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:02.620880 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:02.620999 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:02.625683 (+ 4684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:02.625691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:02.625730 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:02.626272 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:02.626287 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.627881 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.627887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:02.630297 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:02.630316 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.632404 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.632411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:02.632692 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.632721 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 23:37:02.632844 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 23:37:02.633004 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":1,"thread_start_us":830,"threads_started":1}
I20260810 23:37:02.634527 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.620718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.9.48.194:42193)
0810 23:37:02.625015 (+ 4297us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:02.625025 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:02.625059 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:37:02.625196 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:02.625204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:02.625428 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:02.626235 (+ 807us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:02.626450 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.627701 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.630540 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.632137 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.632348 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.632570 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:02.633979 (+ 1409us) server_negotiation.cc:299] Negotiation successful
0810 23:37:02.634176 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4122,"thread_start_us":3986,"threads_started":1}
I20260810 23:37:02.634200 7016 tablet_service.cc:1467] Processing CreateTablet for tablet 601dd01c3f444692a8f171c9aa14a506 (DEFAULT_TABLE table=test-table [id=f20bb9703ad94155906b7b3f5da0406a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:37:02.635962 7016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 601dd01c3f444692a8f171c9aa14a506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:02.635821 6595 tablet_service.cc:1467] Processing CreateTablet for tablet 601dd01c3f444692a8f171c9aa14a506 (DEFAULT_TABLE table=test-table [id=f20bb9703ad94155906b7b3f5da0406a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:37:02.637115 6595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 601dd01c3f444692a8f171c9aa14a506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:02.638348 6809 tablet_service.cc:1467] Processing CreateTablet for tablet 601dd01c3f444692a8f171c9aa14a506 (DEFAULT_TABLE table=test-table [id=f20bb9703ad94155906b7b3f5da0406a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:37:02.639673 6809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 601dd01c3f444692a8f171c9aa14a506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:02.649799 7140 tablet_bootstrap.cc:492] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe: Bootstrap starting.
I20260810 23:37:02.652781 7141 tablet_bootstrap.cc:492] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090: Bootstrap starting.
I20260810 23:37:02.654206 7142 tablet_bootstrap.cc:492] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b: Bootstrap starting.
I20260810 23:37:02.654265 7140 tablet_bootstrap.cc:654] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:02.655746 7140 log.cc:824] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:02.656731 7141 tablet_bootstrap.cc:654] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:02.657464 7142 tablet_bootstrap.cc:654] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:02.658130 7141 log.cc:824] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:02.658716 7142 log.cc:824] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:02.659145 7140 tablet_bootstrap.cc:492] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe: No bootstrap required, opened a new log
I20260810 23:37:02.659482 7140 ts_tablet_manager.cc:1397] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 23:37:02.661103 7141 tablet_bootstrap.cc:492] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090: No bootstrap required, opened a new log
I20260810 23:37:02.661412 7141 ts_tablet_manager.cc:1397] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260810 23:37:02.662982 7142 tablet_bootstrap.cc:492] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b: No bootstrap required, opened a new log
I20260810 23:37:02.663235 7142 ts_tablet_manager.cc:1397] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260810 23:37:02.669874 7141 raft_consensus.cc:348] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b261a06a0b484ee091cb0c8424a6bdbe" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33181 } } peers { permanent_uuid: "3e20c7c22df24fdf9b89f868c4822090" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 40873 } } peers { permanent_uuid: "2477767319ce4919a75f101d8746a59b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42193 } }
I20260810 23:37:02.670300 7141 raft_consensus.cc:374] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:02.670466 7141 raft_consensus.cc:729] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e20c7c22df24fdf9b89f868c4822090, State: Initialized, Role: FOLLOWER
I20260810 23:37:02.670409 7140 raft_consensus.cc:348] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b261a06a0b484ee091cb0c8424a6bdbe" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33181 } } peers { permanent_uuid: "3e20c7c22df24fdf9b89f868c4822090" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 40873 } } peers { permanent_uuid: "2477767319ce4919a75f101d8746a59b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42193 } }
I20260810 23:37:02.670886 7140 raft_consensus.cc:374] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:02.670997 7140 raft_consensus.cc:729] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b261a06a0b484ee091cb0c8424a6bdbe, State: Initialized, Role: FOLLOWER
I20260810 23:37:02.671223 7141 consensus_queue.cc:260] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b261a06a0b484ee091cb0c8424a6bdbe" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33181 } } peers { permanent_uuid: "3e20c7c22df24fdf9b89f868c4822090" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 40873 } } peers { permanent_uuid: "2477767319ce4919a75f101d8746a59b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42193 } }
I20260810 23:37:02.671618 7140 consensus_queue.cc:260] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b261a06a0b484ee091cb0c8424a6bdbe" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33181 } } peers { permanent_uuid: "3e20c7c22df24fdf9b89f868c4822090" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 40873 } } peers { permanent_uuid: "2477767319ce4919a75f101d8746a59b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42193 } }
I20260810 23:37:02.672626 7116 heartbeater.cc:503] Master 127.9.48.254:39893 was elected leader, sending a full tablet report...
I20260810 23:37:02.673457 7141 ts_tablet_manager.cc:1428] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260810 23:37:02.675199 7140 ts_tablet_manager.cc:1428] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260810 23:37:02.674329 7142 raft_consensus.cc:348] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b261a06a0b484ee091cb0c8424a6bdbe" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33181 } } peers { permanent_uuid: "3e20c7c22df24fdf9b89f868c4822090" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 40873 } } peers { permanent_uuid: "2477767319ce4919a75f101d8746a59b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42193 } }
I20260810 23:37:02.675863 7142 raft_consensus.cc:374] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:02.676003 7142 raft_consensus.cc:729] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2477767319ce4919a75f101d8746a59b, State: Initialized, Role: FOLLOWER
I20260810 23:37:02.676585 7142 consensus_queue.cc:260] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b261a06a0b484ee091cb0c8424a6bdbe" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33181 } } peers { permanent_uuid: "3e20c7c22df24fdf9b89f868c4822090" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 40873 } } peers { permanent_uuid: "2477767319ce4919a75f101d8746a59b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42193 } }
I20260810 23:37:02.677917 6902 heartbeater.cc:503] Master 127.9.48.254:39893 was elected leader, sending a full tablet report...
I20260810 23:37:02.678481 7142 ts_tablet_manager.cc:1428] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
W20260810 23:37:02.685971 6700 tablet.cc:2369] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:37:02.702138 7148 raft_consensus.cc:482] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:37:02.702441 7148 raft_consensus.cc:504] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b261a06a0b484ee091cb0c8424a6bdbe" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33181 } } peers { permanent_uuid: "3e20c7c22df24fdf9b89f868c4822090" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 40873 } } peers { permanent_uuid: "2477767319ce4919a75f101d8746a59b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42193 } }
I20260810 23:37:02.704066 7148 leader_election.cc:288] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e20c7c22df24fdf9b89f868c4822090 (127.9.48.195:40873), 2477767319ce4919a75f101d8746a59b (127.9.48.194:42193)
I20260810 23:37:02.712141 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.704502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:52641 (local address 127.9.48.195:40873)
0810 23:37:02.704800 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:02.704810 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:02.705311 (+ 501us) server_negotiation.cc:407] Connection header received
0810 23:37:02.705581 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:02.705592 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:02.705704 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:02.705827 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:02.706561 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.707491 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.710032 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.711025 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.711190 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.711276 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:02.711784 (+ 508us) server_negotiation.cc:299] Negotiation successful
0810 23:37:02.711925 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260810 23:37:02.712406 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.704467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:40873 (local address 127.9.48.193:52641)
0810 23:37:02.705207 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:02.705224 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:02.705400 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:02.705882 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:02.705887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:02.705922 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:02.706411 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:02.706426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.707644 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.707648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:02.709862 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:02.709879 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.711248 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.711254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:02.711567 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.711596 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 23:37:02.711730 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 23:37:02.711875 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":416,"threads_started":1}
I20260810 23:37:02.712608 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.705505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:37243 (local address 127.9.48.194:42193)
0810 23:37:02.705769 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:02.705777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:02.706285 (+ 508us) server_negotiation.cc:407] Connection header received
0810 23:37:02.706540 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:02.706545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:02.706628 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:02.706733 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:02.707409 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.708179 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.710513 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.711593 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.711745 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.711841 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:02.712171 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 23:37:02.712321 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:37:02.712963 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.705361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:42193 (local address 127.9.48.193:37243)
0810 23:37:02.706172 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:02.706189 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:02.706356 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:02.706849 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:02.706853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:02.706879 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:02.707237 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:02.707247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.708328 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.708334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:02.710358 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:02.710370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.711820 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.711826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:02.711983 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.712023 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 23:37:02.712120 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:37:02.712234 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":4,"thread_start_us":408,"threads_started":1}
I20260810 23:37:02.713604 7040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601dd01c3f444692a8f171c9aa14a506" candidate_uuid: "b261a06a0b484ee091cb0c8424a6bdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e20c7c22df24fdf9b89f868c4822090" is_pre_election: true
I20260810 23:37:02.713972 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601dd01c3f444692a8f171c9aa14a506" candidate_uuid: "b261a06a0b484ee091cb0c8424a6bdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2477767319ce4919a75f101d8746a59b" is_pre_election: true
I20260810 23:37:02.714202 7040 raft_consensus.cc:2454] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b261a06a0b484ee091cb0c8424a6bdbe in term 0.
I20260810 23:37:02.714521 6839 raft_consensus.cc:2454] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b261a06a0b484ee091cb0c8424a6bdbe in term 0.
I20260810 23:37:02.714905 6523 leader_election.cc:302] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e20c7c22df24fdf9b89f868c4822090, b261a06a0b484ee091cb0c8424a6bdbe; no voters:
I20260810 23:37:02.715279 7148 raft_consensus.cc:2781] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:37:02.715425 7148 raft_consensus.cc:482] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:37:02.715606 7148 raft_consensus.cc:3037] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:02.718122 7148 raft_consensus.cc:504] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b261a06a0b484ee091cb0c8424a6bdbe" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33181 } } peers { permanent_uuid: "3e20c7c22df24fdf9b89f868c4822090" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 40873 } } peers { permanent_uuid: "2477767319ce4919a75f101d8746a59b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42193 } }
I20260810 23:37:02.718680 7148 leader_election.cc:288] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [CANDIDATE]: Term 1 election: Requested vote from peers 3e20c7c22df24fdf9b89f868c4822090 (127.9.48.195:40873), 2477767319ce4919a75f101d8746a59b (127.9.48.194:42193)
I20260810 23:37:02.719189 7040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601dd01c3f444692a8f171c9aa14a506" candidate_uuid: "b261a06a0b484ee091cb0c8424a6bdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e20c7c22df24fdf9b89f868c4822090"
I20260810 23:37:02.719242 6839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "601dd01c3f444692a8f171c9aa14a506" candidate_uuid: "b261a06a0b484ee091cb0c8424a6bdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2477767319ce4919a75f101d8746a59b"
I20260810 23:37:02.719393 7040 raft_consensus.cc:3037] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:02.719425 6839 raft_consensus.cc:3037] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:02.721680 7040 raft_consensus.cc:2454] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b261a06a0b484ee091cb0c8424a6bdbe in term 1.
I20260810 23:37:02.721715 6839 raft_consensus.cc:2454] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b261a06a0b484ee091cb0c8424a6bdbe in term 1.
I20260810 23:37:02.722105 6523 leader_election.cc:302] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e20c7c22df24fdf9b89f868c4822090, b261a06a0b484ee091cb0c8424a6bdbe; no voters:
I20260810 23:37:02.722404 7148 raft_consensus.cc:2781] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:02.722887 7148 raft_consensus.cc:686] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [term 1 LEADER]: Becoming Leader. State: Replica: b261a06a0b484ee091cb0c8424a6bdbe, State: Running, Role: LEADER
I20260810 23:37:02.723430 7148 consensus_queue.cc:237] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b261a06a0b484ee091cb0c8424a6bdbe" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33181 } } peers { permanent_uuid: "3e20c7c22df24fdf9b89f868c4822090" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 40873 } } peers { permanent_uuid: "2477767319ce4919a75f101d8746a59b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42193 } }
I20260810 23:37:02.727859 6393 catalog_manager.cc:5495] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe reported cstate change: term changed from 0 to 1, leader changed from <none> to b261a06a0b484ee091cb0c8424a6bdbe (127.9.48.193). New cstate: current_term: 1 leader_uuid: "b261a06a0b484ee091cb0c8424a6bdbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b261a06a0b484ee091cb0c8424a6bdbe" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 33181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e20c7c22df24fdf9b89f868c4822090" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 40873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2477767319ce4919a75f101d8746a59b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42193 } health_report { overall_health: UNKNOWN } } }
W20260810 23:37:02.741146 6903 tablet.cc:2369] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:37:02.754545 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.747934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.9.48.193:33181)
0810 23:37:02.748165 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:02.748170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:02.748192 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:37:02.748378 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:02.748382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:02.748467 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:02.748691 (+ 224us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:37:02.749152 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.749976 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.751556 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.752331 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.752473 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.753138 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:37:02.753814 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:37:02.754046 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:02.754167 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 23:37:02.754219 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:37:02.764240 7040 raft_consensus.cc:1261] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090 [term 1 FOLLOWER]: Refusing update from remote peer b261a06a0b484ee091cb0c8424a6bdbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:37:02.764369 6839 raft_consensus.cc:1261] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b [term 1 FOLLOWER]: Refusing update from remote peer b261a06a0b484ee091cb0c8424a6bdbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:37:02.765332 7152 consensus_queue.cc:1035] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e20c7c22df24fdf9b89f868c4822090" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:37:02.765812 7148 consensus_queue.cc:1035] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2477767319ce4919a75f101d8746a59b" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 42193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:37:02.778925 7156 mvcc.cc:205] Tried to move back new op lower bound from 7317114973232025600 to 7317114973079089152. Current Snapshot: MvccSnapshot[applied={T|T < 7317114973232025600}]
I20260810 23:37:02.795301 6393 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:37:02.795686 6393 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:37:02.800685 6393 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 543b227734304239a14c45c246ab8cd5: Sending DeleteTablet for 3 replicas of tablet 601dd01c3f444692a8f171c9aa14a506
I20260810 23:37:02.801987 7016 tablet_service.cc:1514] Processing DeleteTablet for tablet 601dd01c3f444692a8f171c9aa14a506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:37:02 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:39046
I20260810 23:37:02.802441 6809 tablet_service.cc:1514] Processing DeleteTablet for tablet 601dd01c3f444692a8f171c9aa14a506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:37:02 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:41386
I20260810 23:37:02.802501 6808 tablet_service.cc:1514] Processing DeleteTablet for tablet 601dd01c3f444692a8f171c9aa14a506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:37:02 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:41386
I20260810 23:37:02.802729 6595 tablet_service.cc:1514] Processing DeleteTablet for tablet 601dd01c3f444692a8f171c9aa14a506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:37:02 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:54728
I20260810 23:37:02.804560 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 6491
I20260810 23:37:02.826556 7167 tablet_replica.cc:330] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090: stopping tablet replica
I20260810 23:37:02.827167 7167 raft_consensus.cc:2229] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:37:02.827872 7167 raft_consensus.cc:2258] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:37:02.828845 7171 tablet_replica.cc:330] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe: stopping tablet replica
I20260810 23:37:02.829336 7171 raft_consensus.cc:2229] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:37:02.830137 7171 raft_consensus.cc:2258] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:37:02.833773 7170 tablet_replica.cc:330] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b: stopping tablet replica
I20260810 23:37:02.834349 7167 ts_tablet_manager.cc:1905] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:37:02.834379 7170 raft_consensus.cc:2229] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:37:02.835568 7170 raft_consensus.cc:2258] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:37:02.838371 7170 ts_tablet_manager.cc:1905] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:37:02.839408 7171 ts_tablet_manager.cc:1905] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 23:37:02 dist-test-slave-04bb krb5kdc[6325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:37:02.846458 7170 ts_tablet_manager.cc:1918] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:37:02.846783 7170 log.cc:1196] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/601dd01c3f444692a8f171c9aa14a506
I20260810 23:37:02.847505 7171 ts_tablet_manager.cc:1918] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:37:02.847554 7170 ts_tablet_manager.cc:1939] T 601dd01c3f444692a8f171c9aa14a506 P 2477767319ce4919a75f101d8746a59b: Deleting consensus metadata
I20260810 23:37:02.847698 7171 log.cc:1196] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/601dd01c3f444692a8f171c9aa14a506
I20260810 23:37:02.848280 7171 ts_tablet_manager.cc:1939] T 601dd01c3f444692a8f171c9aa14a506 P b261a06a0b484ee091cb0c8424a6bdbe: Deleting consensus metadata
I20260810 23:37:02.849629 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.806343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.9.48.193:33181)
0810 23:37:02.806600 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:02.806612 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:02.822887 (+ 16275us) server_negotiation.cc:407] Connection header received
0810 23:37:02.823493 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:02.823502 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:02.823629 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:02.823724 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:02.830678 (+ 6954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.832061 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.834447 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.835175 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.835333 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.841244 (+ 5911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:02.841265 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:02.841283 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:02.841330 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:02.845784 (+ 4454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:02.846751 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:02.846760 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:02.846764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:02.846934 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:02.847476 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:02.847482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:02.847486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:02.848019 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:02.848336 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:02.848972 (+ 636us) server_negotiation.cc:299] Negotiation successful
0810 23:37:02.849251 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 23:37:02.850503 6374 catalog_manager.cc:4841] TS 2477767319ce4919a75f101d8746a59b (127.9.48.194:42193): tablet 601dd01c3f444692a8f171c9aa14a506 (table test-table [id=f20bb9703ad94155906b7b3f5da0406a]) successfully deleted
I20260810 23:37:02.851155 6691 generic_service.cc:196] Checking for leaks (request via RPC)
W20260810 23:37:02.851917 6374 catalog_manager.cc:4808] TS 2477767319ce4919a75f101d8746a59b (127.9.48.194:42193): delete failed for tablet 601dd01c3f444692a8f171c9aa14a506 because the tablet was not found. No further retry: Not found: Tablet not found: 601dd01c3f444692a8f171c9aa14a506
I20260810 23:37:02.854000 7167 ts_tablet_manager.cc:1918] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:37:02.854285 7167 log.cc:1196] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/601dd01c3f444692a8f171c9aa14a506
I20260810 23:37:02.855014 7167 ts_tablet_manager.cc:1939] T 601dd01c3f444692a8f171c9aa14a506 P 3e20c7c22df24fdf9b89f868c4822090: Deleting consensus metadata
I20260810 23:37:02.856899 6374 catalog_manager.cc:4841] TS 3e20c7c22df24fdf9b89f868c4822090 (127.9.48.195:40873): tablet 601dd01c3f444692a8f171c9aa14a506 (table test-table [id=f20bb9703ad94155906b7b3f5da0406a]) successfully deleted
I20260810 23:37:02.953065 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 6491
I20260810 23:37:02.953786 6374 catalog_manager.cc:4841] TS b261a06a0b484ee091cb0c8424a6bdbe (127.9.48.193:33181): tablet 601dd01c3f444692a8f171c9aa14a506 (table test-table [id=f20bb9703ad94155906b7b3f5da0406a]) successfully deleted
I20260810 23:37:02.982348 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 6711
Aug 10 23:37:02 dist-test-slave-04bb krb5kdc[6325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:37:03.000211 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:02.983153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.9.48.194:42193)
0810 23:37:02.983366 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:02.983371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:02.983392 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:37:02.983521 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:02.983526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:02.983593 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:02.983687 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:02.985104 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.985854 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.987391 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:02.988107 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:02.988254 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:02.992659 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:02.992675 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:02.992692 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:02.992742 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:02.996899 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:02.997790 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:02.997800 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:02.997805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:02.997961 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:02.998384 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:02.998390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:02.998394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:02.998939 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:02.999235 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:02.999514 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 23:37:02.999755 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 23:37:03.001188 6896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:03.092911 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 6711
I20260810 23:37:03.120428 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 6914
Aug 10 23:37:03 dist-test-slave-04bb krb5kdc[6325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:37:03.137511 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:03.121345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.9.48.195:40873)
0810 23:37:03.121555 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:03.121562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:03.121583 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:37:03.121768 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:03.121772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:03.121854 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:03.121964 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:03.123365 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:03.124114 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:03.125581 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:03.125990 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:03.126127 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:03.131343 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:03.131360 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:03.131377 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:03.131424 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:03.134561 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:03.135303 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:03.135311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:03.135313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:03.135452 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:03.135859 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:03.135865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:03.135868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:03.136244 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:03.136573 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:03.136923 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 23:37:03.137174 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:37:03.138198 7103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:03.228859 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 6914
I20260810 23:37:03.255173 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 6348
I20260810 23:37:03.256410 6440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:03.366717 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 6348
2026-08-10T23:37:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4783 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T23:37:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:03Z Disabled control of system clock
I20260810 23:37:03.466177 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:37075
--webserver_interface=127.9.48.254
--webserver_port=0
--builtin_ntp_servers=127.9.48.212:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:37075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:37:03.813429 7184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:03.813750 7184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:03.813833 7184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:03.820945 7184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:03.821050 7184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:03.821089 7184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:03.821139 7184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:03.821192 7184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:03.830235 7184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:37075
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:37075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:03.831523 7184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:03.834172 7184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:03.846412 7196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:03.846493 7204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:03.846464 7199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:03.847417 7184 server_base.cc:1034] running on GCE node
I20260810 23:37:03.848199 7184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:03.849680 7184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:03.850950 7184 hybrid_clock.cc:648] HybridClock initialized: now 1786405023850882 us; error 56 us; skew 500 ppm
I20260810 23:37:03.851377 7184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:37:03.853893 7184 webserver.cc:460] Webserver started at http://127.9.48.254:37577/ using document root <none> and password file <none>
I20260810 23:37:03.854631 7184 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:03.854782 7184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:03.855144 7184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:03.858358 7184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "3a67664e9ac64c33ba55fbe9708dfbc1"
format_stamp: "Formatted at 2026-08-10 23:37:03 on dist-test-slave-04bb"
server_key: "a3744ae492d572dabfd8df319992eb5c"
server_key_iv: "ed2c70a0067bc2a0aa6854d4bd6b7fa4"
server_key_version: "encryptionkey@0"
I20260810 23:37:03.859247 7184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "3a67664e9ac64c33ba55fbe9708dfbc1"
format_stamp: "Formatted at 2026-08-10 23:37:03 on dist-test-slave-04bb"
server_key: "a3744ae492d572dabfd8df319992eb5c"
server_key_iv: "ed2c70a0067bc2a0aa6854d4bd6b7fa4"
server_key_version: "encryptionkey@0"
I20260810 23:37:03.864790 7184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 23:37:03.868604 7215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:03.870615 7184 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:37:03.870822 7184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "3a67664e9ac64c33ba55fbe9708dfbc1"
format_stamp: "Formatted at 2026-08-10 23:37:03 on dist-test-slave-04bb"
server_key: "a3744ae492d572dabfd8df319992eb5c"
server_key_iv: "ed2c70a0067bc2a0aa6854d4bd6b7fa4"
server_key_version: "encryptionkey@0"
I20260810 23:37:03.871059 7184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:03.888549 7184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:03.889544 7184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:03.889869 7184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:03.910277 7184 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:37075
I20260810 23:37:03.910373 7291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:37075 every 8 connection(s)
I20260810 23:37:03.911691 7184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:03.914829 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 7184
I20260810 23:37:03.915051 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:03.915719 9411 external_mini_cluster.cc:1428] Setting key 895e60ceb8ff58f095f2f51bb3b8c176
I20260810 23:37:03.916798 7292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:03.930446 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:03.917869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.9.48.254:37075)
0810 23:37:03.918900 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:03.918911 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:03.918952 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:37:03.920576 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:03.920608 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:03.921077 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:03.921435 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:03.922126 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:03.923471 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:03.925595 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:03.926285 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:03.926428 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:03.927880 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:03.927889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:03.927904 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:03.928488 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:03.928637 (+ 149us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:03.928663 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:03.928765 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:03.929014 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 23:37:03.929409 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":396,"threads_started":1}
I20260810 23:37:03.932289 7292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1: Bootstrap starting.
I20260810 23:37:03.935607 7292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:03.936882 7292 log.cc:824] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:03.939726 7292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1: No bootstrap required, opened a new log
I20260810 23:37:03.947088 7292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a67664e9ac64c33ba55fbe9708dfbc1" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 37075 } }
I20260810 23:37:03.947492 7292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:03.947624 7292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a67664e9ac64c33ba55fbe9708dfbc1, State: Initialized, Role: FOLLOWER
I20260810 23:37:03.948275 7292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a67664e9ac64c33ba55fbe9708dfbc1" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 37075 } }
I20260810 23:37:03.948510 7292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:03.948720 7292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:03.948917 7292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:03.951112 7292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a67664e9ac64c33ba55fbe9708dfbc1" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 37075 } }
I20260810 23:37:03.951668 7292 leader_election.cc:302] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a67664e9ac64c33ba55fbe9708dfbc1; no voters:
I20260810 23:37:03.952258 7292 leader_election.cc:288] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:03.952437 7299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:03.953310 7299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [term 1 LEADER]: Becoming Leader. State: Replica: 3a67664e9ac64c33ba55fbe9708dfbc1, State: Running, Role: LEADER
I20260810 23:37:03.953859 7299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a67664e9ac64c33ba55fbe9708dfbc1" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 37075 } }
I20260810 23:37:03.954510 7292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:03.957623 7302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a67664e9ac64c33ba55fbe9708dfbc1. Latest consensus state: current_term: 1 leader_uuid: "3a67664e9ac64c33ba55fbe9708dfbc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a67664e9ac64c33ba55fbe9708dfbc1" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 37075 } } }
I20260810 23:37:03.958014 7302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:03.957374 7301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a67664e9ac64c33ba55fbe9708dfbc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a67664e9ac64c33ba55fbe9708dfbc1" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 37075 } } }
I20260810 23:37:03.959501 7301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:03.961381 7311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:03.966986 7311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:03.976673 7311 catalog_manager.cc:1347] Generated new cluster ID: 1c2cfc1ef86f440db5ab24e2262ab499
I20260810 23:37:03.976807 7311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:03.988595 7311 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:03.989939 7311 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:04.000315 7311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1: Generated new TSK 0
I20260810 23:37:04.001334 7311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:04.014703 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:37075
--builtin_ntp_servers=127.9.48.212:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:37:04.369482 7325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:04.369750 7325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:04.369908 7325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:04.377209 7325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:04.377300 7325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:04.377488 7325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:37:04.386315 7325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:37075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:04.387517 7325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:04.389271 7325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:04.402470 7332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:04.402420 7339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:04.402439 7334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:04.403436 7325 server_base.cc:1034] running on GCE node
I20260810 23:37:04.403981 7325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:04.404985 7325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:04.406189 7325 hybrid_clock.cc:648] HybridClock initialized: now 1786405024406136 us; error 43 us; skew 500 ppm
I20260810 23:37:04.406533 7325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:37:04.415077 7325 webserver.cc:460] Webserver started at http://127.9.48.193:32847/ using document root <none> and password file <none>
I20260810 23:37:04.415896 7325 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:04.415988 7325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:04.416364 7325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:04.419319 7325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "052dfe96c13f428e9371c1f2d60eeb66"
format_stamp: "Formatted at 2026-08-10 23:37:04 on dist-test-slave-04bb"
server_key: "1e52f47b7aaf96a6ed32285bbafafe13"
server_key_iv: "7b970410e98eca8f5e4dd8b397646085"
server_key_version: "encryptionkey@0"
I20260810 23:37:04.420110 7325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "052dfe96c13f428e9371c1f2d60eeb66"
format_stamp: "Formatted at 2026-08-10 23:37:04 on dist-test-slave-04bb"
server_key: "1e52f47b7aaf96a6ed32285bbafafe13"
server_key_iv: "7b970410e98eca8f5e4dd8b397646085"
server_key_version: "encryptionkey@0"
I20260810 23:37:04.425537 7325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:37:04.429141 7348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:04.430532 7325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260810 23:37:04.430707 7325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "052dfe96c13f428e9371c1f2d60eeb66"
format_stamp: "Formatted at 2026-08-10 23:37:04 on dist-test-slave-04bb"
server_key: "1e52f47b7aaf96a6ed32285bbafafe13"
server_key_iv: "7b970410e98eca8f5e4dd8b397646085"
server_key_version: "encryptionkey@0"
I20260810 23:37:04.430930 7325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:04.450299 7325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:04.451104 7325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:04.451429 7325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:04.452433 7325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:37:04.453990 7325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:37:04.454124 7325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:04.454252 7325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:37:04.454344 7325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:04.489707 7325 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:39951
I20260810 23:37:04.489801 7517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:39951 every 8 connection(s)
I20260810 23:37:04.491051 7325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:37:04.497031 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 7325
I20260810 23:37:04.497222 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:37:04.497690 9411 external_mini_cluster.cc:1428] Setting key 3478de515085bc8cc718027190d0d439
I20260810 23:37:04.500859 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:37075
--builtin_ntp_servers=127.9.48.212:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:37:04.505611 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:04.493945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:45129 (local address 127.9.48.254:37075)
0810 23:37:04.494461 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:04.494466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:04.495697 (+ 1231us) server_negotiation.cc:407] Connection header received
0810 23:37:04.497916 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:04.497923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:04.498028 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:04.498162 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:04.499573 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:04.500569 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:04.502468 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:04.502972 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:04.503121 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:04.504667 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:04.504673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:04.504683 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:04.504703 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:04.504744 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:04.504758 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:04.504866 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:04.505122 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 23:37:04.505228 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":276,"threads_started":1}
I20260810 23:37:04.506707 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:04.494551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:37075 (local address 127.9.48.193:45129)
0810 23:37:04.495501 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:04.495554 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:04.497581 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:04.498418 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:04.498427 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:04.498519 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:04.499362 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:04.499390 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:04.500755 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:04.500786 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:04.502291 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:04.502306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:04.503232 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:04.503241 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:04.503492 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:04.504279 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:37:04.504282 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:04.504325 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:37:04.504413 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:04.504866 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:04.504882 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:04.504893 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 23:37:04.505108 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 23:37:04.505548 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":294,"threads_started":1}
I20260810 23:37:04.508630 7520 heartbeater.cc:347] Connected to a master server at 127.9.48.254:37075
I20260810 23:37:04.508960 7520 heartbeater.cc:464] Registering TS with master...
I20260810 23:37:04.509747 7520 heartbeater.cc:511] Master 127.9.48.254:37075 requested a full tablet report, sending...
I20260810 23:37:04.512146 7239 ts_manager.cc:194] Registered new tserver with Master: 052dfe96c13f428e9371c1f2d60eeb66 (127.9.48.193:39951)
I20260810 23:37:04.514271 7239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.48.193:45129
W20260810 23:37:04.850389 7525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:04.850733 7525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:04.850899 7525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:04.858682 7525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:04.858829 7525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:04.859066 7525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:37:04.868474 7525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:37075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:04.869865 7525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:04.871784 7525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:04.886147 7540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:04.886395 7535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:04.886389 7538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:04.887488 7525 server_base.cc:1034] running on GCE node
I20260810 23:37:04.888285 7525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:04.889489 7525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:04.890779 7525 hybrid_clock.cc:648] HybridClock initialized: now 1786405024890725 us; error 55 us; skew 500 ppm
I20260810 23:37:04.891166 7525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:37:04.893844 7525 webserver.cc:460] Webserver started at http://127.9.48.194:34249/ using document root <none> and password file <none>
I20260810 23:37:04.894793 7525 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:04.894975 7525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:04.895362 7525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:04.898658 7525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "004f4a5c51be4ad8a636c95490bc6865"
format_stamp: "Formatted at 2026-08-10 23:37:04 on dist-test-slave-04bb"
server_key: "1a00ed9e160b964915fdaaa193c006a5"
server_key_iv: "a7ce522395e9991f9db5775bdbde90c9"
server_key_version: "encryptionkey@0"
I20260810 23:37:04.899555 7525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "004f4a5c51be4ad8a636c95490bc6865"
format_stamp: "Formatted at 2026-08-10 23:37:04 on dist-test-slave-04bb"
server_key: "1a00ed9e160b964915fdaaa193c006a5"
server_key_iv: "a7ce522395e9991f9db5775bdbde90c9"
server_key_version: "encryptionkey@0"
I20260810 23:37:04.905581 7525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 23:37:04.909631 7546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:04.911170 7525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:37:04.911352 7525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "004f4a5c51be4ad8a636c95490bc6865"
format_stamp: "Formatted at 2026-08-10 23:37:04 on dist-test-slave-04bb"
server_key: "1a00ed9e160b964915fdaaa193c006a5"
server_key_iv: "a7ce522395e9991f9db5775bdbde90c9"
server_key_version: "encryptionkey@0"
I20260810 23:37:04.911584 7525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:04.947054 7525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:04.948009 7525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:04.948352 7525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:04.949451 7525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:37:04.951031 7525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:37:04.951162 7525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:04.951297 7525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:37:04.951402 7525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:04.987726 7525 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:34435
I20260810 23:37:04.987816 7723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:34435 every 8 connection(s)
I20260810 23:37:04.989204 7525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:37:04.995524 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 7525
I20260810 23:37:04.995733 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:37:04.996232 9411 external_mini_cluster.cc:1428] Setting key 302ac7b43c21bc633fd7808bb9ea2c8f
I20260810 23:37:04.999353 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:37075
--builtin_ntp_servers=127.9.48.212:38015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:37:05.002655 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:04.991935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:56985 (local address 127.9.48.254:37075)
0810 23:37:04.992128 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:04.992134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:04.993739 (+ 1605us) server_negotiation.cc:407] Connection header received
0810 23:37:04.995584 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:04.995591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:04.995706 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:04.995834 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:04.997051 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:04.998141 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:04.999779 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.000179 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.000315 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.001796 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:05.001809 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:05.001813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:05.001831 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:05.001864 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:05.001876 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:05.001975 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:05.002221 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 23:37:05.002324 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:37:05.003721 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:04.992493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:37075 (local address 127.9.48.194:56985)
0810 23:37:04.993575 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:04.993619 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:04.995299 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:04.996075 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:04.996084 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:04.996181 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:04.996874 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:04.996892 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:04.998308 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:04.998312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:04.999626 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:04.999640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.000407 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.000416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:05.000717 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.001457 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:37:05.001460 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:05.001501 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:37:05.001588 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:05.001977 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:05.001991 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:05.002001 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 23:37:05.002193 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 23:37:05.002607 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":328,"threads_started":1}
I20260810 23:37:05.005280 7724 heartbeater.cc:347] Connected to a master server at 127.9.48.254:37075
I20260810 23:37:05.005609 7724 heartbeater.cc:464] Registering TS with master...
I20260810 23:37:05.006348 7724 heartbeater.cc:511] Master 127.9.48.254:37075 requested a full tablet report, sending...
I20260810 23:37:05.007916 7239 ts_manager.cc:194] Registered new tserver with Master: 004f4a5c51be4ad8a636c95490bc6865 (127.9.48.194:34435)
I20260810 23:37:05.009164 7239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.48.194:56985
W20260810 23:37:05.350037 7729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:05.350332 7729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:05.350425 7729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:05.357604 7729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:05.357703 7729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:05.357810 7729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:37:05.366854 7729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:38015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:37075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:05.368144 7729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:05.369928 7729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:05.383598 7738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:05.383643 7739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:05.383682 7742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:05.385061 7729 server_base.cc:1034] running on GCE node
I20260810 23:37:05.385746 7729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:05.386786 7729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:05.388038 7729 hybrid_clock.cc:648] HybridClock initialized: now 1786405025387971 us; error 60 us; skew 500 ppm
I20260810 23:37:05.388383 7729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:37:05.390704 7729 webserver.cc:460] Webserver started at http://127.9.48.195:39721/ using document root <none> and password file <none>
I20260810 23:37:05.391417 7729 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:05.391535 7729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:05.391886 7729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:05.394804 7729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "b7f4c4d5643141f9b298a20ab1a0ccce"
format_stamp: "Formatted at 2026-08-10 23:37:05 on dist-test-slave-04bb"
server_key: "887acc5a49e8c708d210c91173b94c4d"
server_key_iv: "c18dd4d38c0f468c2e5541baaefada43"
server_key_version: "encryptionkey@0"
I20260810 23:37:05.395570 7729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "b7f4c4d5643141f9b298a20ab1a0ccce"
format_stamp: "Formatted at 2026-08-10 23:37:05 on dist-test-slave-04bb"
server_key: "887acc5a49e8c708d210c91173b94c4d"
server_key_iv: "c18dd4d38c0f468c2e5541baaefada43"
server_key_version: "encryptionkey@0"
I20260810 23:37:05.400996 7729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 23:37:05.404258 7751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:05.405609 7729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:37:05.405799 7729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "b7f4c4d5643141f9b298a20ab1a0ccce"
format_stamp: "Formatted at 2026-08-10 23:37:05 on dist-test-slave-04bb"
server_key: "887acc5a49e8c708d210c91173b94c4d"
server_key_iv: "c18dd4d38c0f468c2e5541baaefada43"
server_key_version: "encryptionkey@0"
I20260810 23:37:05.406028 7729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:05.422569 7729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:05.423386 7729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:05.423735 7729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:05.424759 7729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:37:05.426214 7729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:37:05.426353 7729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:05.426476 7729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:37:05.426564 7729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:05.461727 7729 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:33819
I20260810 23:37:05.461817 7927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:33819 every 8 connection(s)
I20260810 23:37:05.463083 7729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:37:05.464447 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 7729
I20260810 23:37:05.464640 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:37:05.465251 9411 external_mini_cluster.cc:1428] Setting key a250e67063c2ed22f83ae33b59936667
I20260810 23:37:05.476429 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:05.466116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:38455 (local address 127.9.48.254:37075)
0810 23:37:05.466362 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:05.466372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:05.468096 (+ 1724us) server_negotiation.cc:407] Connection header received
0810 23:37:05.470032 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:05.470036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:05.470148 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:05.470236 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:05.471279 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.472011 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.473575 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.474003 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.474106 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.475616 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:05.475629 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:05.475633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:05.475650 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:05.475681 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:05.475693 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:05.475764 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:05.475975 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 23:37:05.476048 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260810 23:37:05.477371 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:05.466787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:37075 (local address 127.9.48.195:38455)
0810 23:37:05.467923 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:05.467973 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:05.469776 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:05.470441 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:05.470448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:05.470531 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:05.471099 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:05.471115 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.472146 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.472149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:05.473439 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:05.473451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.474239 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.474246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:05.474440 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.475268 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:37:05.475270 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:05.475334 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:37:05.475432 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:05.475788 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:05.475797 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:05.475803 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:37:05.475946 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 23:37:05.476312 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":341,"threads_started":1}
I20260810 23:37:05.478650 7928 heartbeater.cc:347] Connected to a master server at 127.9.48.254:37075
I20260810 23:37:05.478981 7928 heartbeater.cc:464] Registering TS with master...
I20260810 23:37:05.479792 7928 heartbeater.cc:511] Master 127.9.48.254:37075 requested a full tablet report, sending...
I20260810 23:37:05.481289 7239 ts_manager.cc:194] Registered new tserver with Master: b7f4c4d5643141f9b298a20ab1a0ccce (127.9.48.195:33819)
I20260810 23:37:05.482090 7239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.48.195:38455
I20260810 23:37:05.493525 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:37:05.503928 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:05.497703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.9.48.254:37075)
0810 23:37:05.497902 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:05.497906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:05.498372 (+ 466us) server_negotiation.cc:407] Connection header received
0810 23:37:05.498505 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:05.498509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:05.498574 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:05.498672 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:05.499165 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.499844 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.501377 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.501808 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.501949 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.503157 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:05.503171 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:05.503174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:05.503193 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:05.503224 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:05.503237 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:05.503402 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:05.503610 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 23:37:05.503678 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:37:05.508006 7239 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:37:05.515053 7239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:37:05.517982 7520 heartbeater.cc:503] Master 127.9.48.254:37075 was elected leader, sending a full tablet report...
I20260810 23:37:05.539779 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:05.527787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:33819 (local address 127.0.0.1:36344)
0810 23:37:05.530021 (+ 2234us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:05.530039 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:05.530261 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:05.531190 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:05.531195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:05.531218 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:05.531666 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:05.531681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.533660 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.533666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:05.537009 (+ 3343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:05.537021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.538934 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.538944 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:05.539104 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.539127 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:37:05.539261 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 23:37:05.539402 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2106,"mutex_wait_us":1136,"thread_start_us":1935,"threads_started":1}
I20260810 23:37:05.539877 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:05.528421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.9.48.195:33819)
0810 23:37:05.529590 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:05.529600 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:05.530157 (+ 557us) server_negotiation.cc:407] Connection header received
0810 23:37:05.530456 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:05.530467 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:05.530672 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:05.530848 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:05.531877 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.533494 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.537185 (+ 3691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.538513 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.538654 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.538964 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:05.539334 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 23:37:05.539500 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":979,"thread_start_us":827,"threads_started":1}
I20260810 23:37:05.539848 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:05.527668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:39951 (local address 127.0.0.1:36920)
0810 23:37:05.528693 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:05.528737 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:05.528902 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:05.530592 (+ 1690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:05.530621 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:05.530658 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:05.531208 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:05.531219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.533661 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.533666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:05.536376 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:05.536392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.538538 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.538544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:05.538820 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.538881 (+ 61us) client_negotiation.cc:769] Sending connection context
0810 23:37:05.539008 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 23:37:05.539202 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":4,"thread_start_us":908,"threads_started":1}
I20260810 23:37:05.541481 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:05.527937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.9.48.193:39951)
0810 23:37:05.529934 (+ 1997us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:05.529943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:05.529972 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:37:05.530086 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:05.530097 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:05.530362 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:05.531456 (+ 1094us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:05.531715 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.533480 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.536803 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.538313 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.539986 (+ 1673us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.540804 (+ 818us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:05.540885 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 23:37:05.541095 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1815,"thread_start_us":1661,"threads_started":1}
I20260810 23:37:05.544212 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:05.530110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:34435 (local address 127.0.0.1:32972)
0810 23:37:05.534754 (+ 4644us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:05.534771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:05.534884 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:05.536001 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:05.536006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:05.536033 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:05.536459 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:05.536471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.538464 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.538471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:05.540909 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:05.540924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.543239 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.543249 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:05.543531 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.543599 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 23:37:05.543695 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 23:37:05.543844 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4517,"mutex_wait_us":3,"thread_start_us":4365,"threads_started":1}
I20260810 23:37:05.544399 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:05.530445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32972 (local address 127.9.48.194:34435)
0810 23:37:05.535431 (+ 4986us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:05.535439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:05.535468 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:37:05.535554 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:05.535561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:05.535775 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:05.535983 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:05.536739 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.538289 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.541119 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.542964 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.543106 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.543359 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:05.543831 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 23:37:05.544002 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4766,"thread_start_us":4639,"threads_started":1}
I20260810 23:37:05.544588 7824 tablet_service.cc:1467] Processing CreateTablet for tablet e74188acaa9f452b88f9b433852f797f (DEFAULT_TABLE table=test-table [id=0b043bfd1e7149dd8d9f5aa94830f659]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:37:05.546335 7824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74188acaa9f452b88f9b433852f797f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:05.546783 7428 tablet_service.cc:1467] Processing CreateTablet for tablet e74188acaa9f452b88f9b433852f797f (DEFAULT_TABLE table=test-table [id=0b043bfd1e7149dd8d9f5aa94830f659]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:37:05.547686 7622 tablet_service.cc:1467] Processing CreateTablet for tablet e74188acaa9f452b88f9b433852f797f (DEFAULT_TABLE table=test-table [id=0b043bfd1e7149dd8d9f5aa94830f659]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:37:05.548277 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74188acaa9f452b88f9b433852f797f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:05.548892 7622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74188acaa9f452b88f9b433852f797f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:05.560727 7967 tablet_bootstrap.cc:492] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865: Bootstrap starting.
I20260810 23:37:05.563136 7969 tablet_bootstrap.cc:492] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66: Bootstrap starting.
I20260810 23:37:05.564808 7968 tablet_bootstrap.cc:492] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce: Bootstrap starting.
I20260810 23:37:05.566272 7967 tablet_bootstrap.cc:654] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:05.566951 7969 tablet_bootstrap.cc:654] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:05.567724 7968 tablet_bootstrap.cc:654] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:05.567680 7967 log.cc:824] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:05.568190 7969 log.cc:824] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:05.568940 7968 log.cc:824] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:05.571008 7969 tablet_bootstrap.cc:492] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66: No bootstrap required, opened a new log
I20260810 23:37:05.571324 7969 ts_tablet_manager.cc:1397] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260810 23:37:05.571376 7967 tablet_bootstrap.cc:492] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865: No bootstrap required, opened a new log
I20260810 23:37:05.571683 7967 ts_tablet_manager.cc:1397] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260810 23:37:05.573695 7968 tablet_bootstrap.cc:492] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce: No bootstrap required, opened a new log
I20260810 23:37:05.574003 7968 ts_tablet_manager.cc:1397] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260810 23:37:05.579756 7969 raft_consensus.cc:348] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dfe96c13f428e9371c1f2d60eeb66" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 39951 } } peers { permanent_uuid: "b7f4c4d5643141f9b298a20ab1a0ccce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33819 } } peers { permanent_uuid: "004f4a5c51be4ad8a636c95490bc6865" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34435 } }
I20260810 23:37:05.580139 7969 raft_consensus.cc:374] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:05.580255 7969 raft_consensus.cc:729] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 052dfe96c13f428e9371c1f2d60eeb66, State: Initialized, Role: FOLLOWER
I20260810 23:37:05.580905 7969 consensus_queue.cc:260] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dfe96c13f428e9371c1f2d60eeb66" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 39951 } } peers { permanent_uuid: "b7f4c4d5643141f9b298a20ab1a0ccce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33819 } } peers { permanent_uuid: "004f4a5c51be4ad8a636c95490bc6865" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34435 } }
I20260810 23:37:05.582712 7969 ts_tablet_manager.cc:1428] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66: Time spent starting tablet: real 0.011s user 0.003s sys 0.007s
I20260810 23:37:05.582433 7967 raft_consensus.cc:348] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dfe96c13f428e9371c1f2d60eeb66" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 39951 } } peers { permanent_uuid: "b7f4c4d5643141f9b298a20ab1a0ccce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33819 } } peers { permanent_uuid: "004f4a5c51be4ad8a636c95490bc6865" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34435 } }
I20260810 23:37:05.582855 7967 raft_consensus.cc:374] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:05.583011 7967 raft_consensus.cc:729] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004f4a5c51be4ad8a636c95490bc6865, State: Initialized, Role: FOLLOWER
I20260810 23:37:05.583668 7967 consensus_queue.cc:260] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dfe96c13f428e9371c1f2d60eeb66" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 39951 } } peers { permanent_uuid: "b7f4c4d5643141f9b298a20ab1a0ccce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33819 } } peers { permanent_uuid: "004f4a5c51be4ad8a636c95490bc6865" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34435 } }
I20260810 23:37:05.585050 7968 raft_consensus.cc:348] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dfe96c13f428e9371c1f2d60eeb66" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 39951 } } peers { permanent_uuid: "b7f4c4d5643141f9b298a20ab1a0ccce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33819 } } peers { permanent_uuid: "004f4a5c51be4ad8a636c95490bc6865" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34435 } }
I20260810 23:37:05.585534 7968 raft_consensus.cc:374] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:05.585664 7724 heartbeater.cc:503] Master 127.9.48.254:37075 was elected leader, sending a full tablet report...
I20260810 23:37:05.585664 7968 raft_consensus.cc:729] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f4c4d5643141f9b298a20ab1a0ccce, State: Initialized, Role: FOLLOWER
I20260810 23:37:05.586248 7968 consensus_queue.cc:260] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dfe96c13f428e9371c1f2d60eeb66" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 39951 } } peers { permanent_uuid: "b7f4c4d5643141f9b298a20ab1a0ccce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33819 } } peers { permanent_uuid: "004f4a5c51be4ad8a636c95490bc6865" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34435 } }
I20260810 23:37:05.587363 7928 heartbeater.cc:503] Master 127.9.48.254:37075 was elected leader, sending a full tablet report...
I20260810 23:37:05.588155 7968 ts_tablet_manager.cc:1428] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 23:37:05.589260 7967 ts_tablet_manager.cc:1428] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865: Time spent starting tablet: real 0.017s user 0.012s sys 0.001s
W20260810 23:37:05.715271 7929 tablet.cc:2369] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:37:05.741514 7726 tablet.cc:2369] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:37:05.743441 7522 tablet.cc:2369] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:37:05.824784 7974 raft_consensus.cc:482] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:37:05.825111 7974 raft_consensus.cc:504] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dfe96c13f428e9371c1f2d60eeb66" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 39951 } } peers { permanent_uuid: "b7f4c4d5643141f9b298a20ab1a0ccce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33819 } } peers { permanent_uuid: "004f4a5c51be4ad8a636c95490bc6865" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34435 } }
I20260810 23:37:05.826974 7974 leader_election.cc:288] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 052dfe96c13f428e9371c1f2d60eeb66 (127.9.48.193:39951), b7f4c4d5643141f9b298a20ab1a0ccce (127.9.48.195:33819)
I20260810 23:37:05.834782 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:05.827664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:38289 (local address 127.9.48.193:39951)
0810 23:37:05.827945 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:05.827954 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:05.828923 (+ 969us) server_negotiation.cc:407] Connection header received
0810 23:37:05.829053 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:05.829061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:05.829165 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:05.829271 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:05.830206 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.830898 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.833048 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.833845 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.834035 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.834104 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:05.834397 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 23:37:05.834543 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 23:37:05.834836 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:05.827648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:51519 (local address 127.9.48.195:33819)
0810 23:37:05.827923 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:05.827933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:05.828737 (+ 804us) server_negotiation.cc:407] Connection header received
0810 23:37:05.829059 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:05.829065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:05.829137 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:05.829191 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:05.830152 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.830890 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.832992 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.833825 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.833983 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.834067 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:05.834448 (+ 381us) server_negotiation.cc:299] Negotiation successful
0810 23:37:05.834573 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 23:37:05.834899 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:05.827391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:33819 (local address 127.9.48.194:51519)
0810 23:37:05.828604 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:05.828626 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:05.828877 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:05.829412 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:05.829418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:05.829443 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:05.829985 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:05.829999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.831028 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.831035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:05.832797 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:05.832811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.834042 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.834049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:05.834267 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.834299 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:37:05.834401 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:37:05.834528 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":101,"thread_start_us":1069,"threads_started":1}
I20260810 23:37:05.835005 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:05.827404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:39951 (local address 127.9.48.194:38289)
0810 23:37:05.828800 (+ 1396us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:05.828818 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:05.828928 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:05.829320 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:05.829328 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:05.829367 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:05.829991 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:05.829999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.831091 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.831095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:05.832825 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:05.832838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.834058 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.834065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:05.834227 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.834261 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:37:05.834334 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 23:37:05.834490 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1275,"mutex_wait_us":148,"spinlock_wait_cycles":44544,"thread_start_us":523,"threads_started":1}
I20260810 23:37:05.835906 7858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74188acaa9f452b88f9b433852f797f" candidate_uuid: "004f4a5c51be4ad8a636c95490bc6865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f4c4d5643141f9b298a20ab1a0ccce" is_pre_election: true
I20260810 23:37:05.836369 7460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74188acaa9f452b88f9b433852f797f" candidate_uuid: "004f4a5c51be4ad8a636c95490bc6865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "052dfe96c13f428e9371c1f2d60eeb66" is_pre_election: true
I20260810 23:37:05.836596 7858 raft_consensus.cc:2454] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 004f4a5c51be4ad8a636c95490bc6865 in term 0.
I20260810 23:37:05.836947 7460 raft_consensus.cc:2454] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 004f4a5c51be4ad8a636c95490bc6865 in term 0.
I20260810 23:37:05.837385 7557 leader_election.cc:302] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 004f4a5c51be4ad8a636c95490bc6865, b7f4c4d5643141f9b298a20ab1a0ccce; no voters:
I20260810 23:37:05.837855 7974 raft_consensus.cc:2781] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:37:05.837975 7974 raft_consensus.cc:482] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:37:05.838145 7974 raft_consensus.cc:3037] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:05.841140 7974 raft_consensus.cc:504] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dfe96c13f428e9371c1f2d60eeb66" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 39951 } } peers { permanent_uuid: "b7f4c4d5643141f9b298a20ab1a0ccce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33819 } } peers { permanent_uuid: "004f4a5c51be4ad8a636c95490bc6865" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34435 } }
I20260810 23:37:05.841878 7974 leader_election.cc:288] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [CANDIDATE]: Term 1 election: Requested vote from peers 052dfe96c13f428e9371c1f2d60eeb66 (127.9.48.193:39951), b7f4c4d5643141f9b298a20ab1a0ccce (127.9.48.195:33819)
I20260810 23:37:05.842275 7460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74188acaa9f452b88f9b433852f797f" candidate_uuid: "004f4a5c51be4ad8a636c95490bc6865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "052dfe96c13f428e9371c1f2d60eeb66"
I20260810 23:37:05.842489 7460 raft_consensus.cc:3037] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:05.842842 7858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74188acaa9f452b88f9b433852f797f" candidate_uuid: "004f4a5c51be4ad8a636c95490bc6865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f4c4d5643141f9b298a20ab1a0ccce"
I20260810 23:37:05.843071 7858 raft_consensus.cc:3037] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:05.845777 7460 raft_consensus.cc:2454] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 004f4a5c51be4ad8a636c95490bc6865 in term 1.
I20260810 23:37:05.846199 7550 leader_election.cc:302] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 004f4a5c51be4ad8a636c95490bc6865, 052dfe96c13f428e9371c1f2d60eeb66; no voters:
I20260810 23:37:05.846269 7858 raft_consensus.cc:2454] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 004f4a5c51be4ad8a636c95490bc6865 in term 1.
I20260810 23:37:05.846544 7974 raft_consensus.cc:2781] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:05.847100 7974 raft_consensus.cc:686] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [term 1 LEADER]: Becoming Leader. State: Replica: 004f4a5c51be4ad8a636c95490bc6865, State: Running, Role: LEADER
I20260810 23:37:05.847604 7974 consensus_queue.cc:237] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dfe96c13f428e9371c1f2d60eeb66" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 39951 } } peers { permanent_uuid: "b7f4c4d5643141f9b298a20ab1a0ccce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33819 } } peers { permanent_uuid: "004f4a5c51be4ad8a636c95490bc6865" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34435 } }
I20260810 23:37:05.852891 7239 catalog_manager.cc:5495] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 004f4a5c51be4ad8a636c95490bc6865 (127.9.48.194). New cstate: current_term: 1 leader_uuid: "004f4a5c51be4ad8a636c95490bc6865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dfe96c13f428e9371c1f2d60eeb66" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 39951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f4c4d5643141f9b298a20ab1a0ccce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "004f4a5c51be4ad8a636c95490bc6865" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 34435 } health_report { overall_health: HEALTHY } } }
I20260810 23:37:05.914763 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:05.908238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32984 (local address 127.9.48.194:34435)
0810 23:37:05.908461 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:05.908469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:05.908508 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:37:05.908641 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:05.908645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:05.908761 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:05.908937 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:37:05.909376 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.910170 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.912257 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.912898 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.913009 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.913598 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:37:05.914089 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:37:05.914276 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:05.914422 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 23:37:05.914458 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 23:37:05.923575 7858 raft_consensus.cc:1261] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce [term 1 FOLLOWER]: Refusing update from remote peer 004f4a5c51be4ad8a636c95490bc6865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:37:05.923619 7460 raft_consensus.cc:1261] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66 [term 1 FOLLOWER]: Refusing update from remote peer 004f4a5c51be4ad8a636c95490bc6865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:37:05.924656 7974 consensus_queue.cc:1035] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "052dfe96c13f428e9371c1f2d60eeb66" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 39951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:37:05.925025 7981 consensus_queue.cc:1035] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f4c4d5643141f9b298a20ab1a0ccce" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 33819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:37:05.937242 7984 mvcc.cc:205] Tried to move back new op lower bound from 7317114986171686912 to 7317114985875877888. Current Snapshot: MvccSnapshot[applied={T|T < 7317114986171686912}]
I20260810 23:37:05.938284 7986 mvcc.cc:205] Tried to move back new op lower bound from 7317114986171686912 to 7317114985875877888. Current Snapshot: MvccSnapshot[applied={T|T < 7317114986171686912}]
I20260810 23:37:05.950223 7239 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:37:05.950519 7239 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:37:05.954485 7239 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a67664e9ac64c33ba55fbe9708dfbc1: Sending DeleteTablet for 3 replicas of tablet e74188acaa9f452b88f9b433852f797f
I20260810 23:37:05.955720 7428 tablet_service.cc:1514] Processing DeleteTablet for tablet e74188acaa9f452b88f9b433852f797f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:37:05 UTC) from {username='slave'} at 127.0.0.1:36920
I20260810 23:37:05.955806 7824 tablet_service.cc:1514] Processing DeleteTablet for tablet e74188acaa9f452b88f9b433852f797f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:37:05 UTC) from {username='slave'} at 127.0.0.1:36344
I20260810 23:37:05.956516 7622 tablet_service.cc:1514] Processing DeleteTablet for tablet e74188acaa9f452b88f9b433852f797f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:37:05 UTC) from {username='slave'} at 127.0.0.1:32972
I20260810 23:37:05.957772 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 7325
I20260810 23:37:05.965942 7998 tablet_replica.cc:330] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865: stopping tablet replica
I20260810 23:37:05.967299 7998 raft_consensus.cc:2229] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:37:05.968914 7998 raft_consensus.cc:2258] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:37:05.974364 7998 ts_tablet_manager.cc:1905] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:37:05.977766 7997 tablet_replica.cc:330] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66: stopping tablet replica
I20260810 23:37:05.978340 7997 raft_consensus.cc:2229] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:37:05.978960 7997 raft_consensus.cc:2258] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:37:05.983482 7997 ts_tablet_manager.cc:1905] T e74188acaa9f452b88f9b433852f797f P 052dfe96c13f428e9371c1f2d60eeb66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:37:05.985535 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:05.959308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.9.48.193:39951)
0810 23:37:05.959491 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:05.959497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:05.972370 (+ 12873us) server_negotiation.cc:407] Connection header received
0810 23:37:05.972895 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:05.972902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:05.973050 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:05.973133 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:05.973843 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.975069 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.979177 (+ 4108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:05.979867 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:05.979974 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:05.983762 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:05.983775 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:05.983792 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:05.984610 (+ 818us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:05.984675 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:05.984701 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:05.984792 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:05.985055 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 23:37:05.985176 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:37:05.986960 7512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:05.987650 7995 tablet_replica.cc:330] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce: stopping tablet replica
I20260810 23:37:05.988169 7995 raft_consensus.cc:2229] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:37:05.988762 7995 raft_consensus.cc:2258] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:37:05.990953 7995 ts_tablet_manager.cc:1905] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:37:05.992844 7998 ts_tablet_manager.cc:1918] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:37:05.993381 7998 log.cc:1196] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/e74188acaa9f452b88f9b433852f797f
I20260810 23:37:05.994158 7998 ts_tablet_manager.cc:1939] T e74188acaa9f452b88f9b433852f797f P 004f4a5c51be4ad8a636c95490bc6865: Deleting consensus metadata
I20260810 23:37:05.996744 7220 catalog_manager.cc:4841] TS 004f4a5c51be4ad8a636c95490bc6865 (127.9.48.194:34435): tablet e74188acaa9f452b88f9b433852f797f (table test-table [id=0b043bfd1e7149dd8d9f5aa94830f659]) successfully deleted
I20260810 23:37:05.998132 7995 ts_tablet_manager.cc:1918] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:37:05.998349 7995 log.cc:1196] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/e74188acaa9f452b88f9b433852f797f
I20260810 23:37:05.998940 7995 ts_tablet_manager.cc:1939] T e74188acaa9f452b88f9b433852f797f P b7f4c4d5643141f9b298a20ab1a0ccce: Deleting consensus metadata
I20260810 23:37:06.000952 7220 catalog_manager.cc:4841] TS b7f4c4d5643141f9b298a20ab1a0ccce (127.9.48.195:33819): tablet e74188acaa9f452b88f9b433852f797f (table test-table [id=0b043bfd1e7149dd8d9f5aa94830f659]) successfully deleted
I20260810 23:37:06.109643 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 7325
I20260810 23:37:06.136883 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 7525
W20260810 23:37:06.136912 7218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.48.193:39951 (error 108)
I20260810 23:37:06.139400 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:06.138523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:39951 (local address 127.0.0.1:36936)
0810 23:37:06.138969 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:06.139170 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":215,"threads_started":1}
W20260810 23:37:06.139760 7218 catalog_manager.cc:4584] TS 052dfe96c13f428e9371c1f2d60eeb66 (127.9.48.193:39951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e74188acaa9f452b88f9b433852f797f: Network error: Client connection negotiation failed: client connection to 127.9.48.193:39951: connect: Connection refused (error 111)
I20260810 23:37:06.145795 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:06.138194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.9.48.194:34435)
0810 23:37:06.138433 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:06.138442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:06.138466 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:37:06.138533 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:06.138537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:06.138621 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:06.138759 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:06.139430 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:06.140656 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:06.142059 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:06.142740 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:06.142892 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:06.143962 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:06.143973 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:06.143991 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:06.144942 (+ 951us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:06.144998 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:06.145022 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:06.145131 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:06.145309 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 23:37:06.145415 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 23:37:06.146648 7715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:06.193570 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:06.193096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:39951 (local address 127.0.0.1:36946)
0810 23:37:06.193327 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:06.193420 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 23:37:06.194705 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:06.194326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:39951 (local address 127.0.0.1:36954)
0810 23:37:06.194483 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:06.194555 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:37:06.255407 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:06.254844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:39951 (local address 127.0.0.1:36962)
0810 23:37:06.255106 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:06.255201 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260810 23:37:06.256603 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:06.256161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:39951 (local address 127.0.0.1:36974)
0810 23:37:06.256322 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:06.256403 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 23:37:06.262607 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 7525
I20260810 23:37:06.293061 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 7729
I20260810 23:37:06.300377 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:06.293880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.9.48.195:33819)
0810 23:37:06.294074 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:06.294081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:06.294167 (+ 86us) server_negotiation.cc:407] Connection header received
0810 23:37:06.294409 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:06.294415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:06.294510 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:06.294636 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:06.295054 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:06.295791 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:06.297212 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:06.297654 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:06.297800 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:06.298946 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:06.298956 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:06.298970 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:06.299573 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:06.299621 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:06.299644 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:06.299781 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:06.299914 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 23:37:06.300024 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:37:06.301231 7920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:06.337380 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:06.336887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:39951 (local address 127.0.0.1:36986)
0810 23:37:06.337117 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:06.337219 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 23:37:06.338500 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:06.338081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:39951 (local address 127.0.0.1:37000)
0810 23:37:06.338289 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:06.338366 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260810 23:37:06.391830 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 7729
I20260810 23:37:06.419008 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 7184
I20260810 23:37:06.420213 7282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:06.529291 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:06.528486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:39951 (local address 127.0.0.1:37002)
0810 23:37:06.528887 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:06.529033 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:39951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232}
I20260810 23:37:06.529829 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 7184
2026-08-10T23:37:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3161 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:06 dist-test-slave-04bb krb5kdc[8011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:06 dist-test-slave-04bb krb5kdc[8011](info): set up 2 sockets
Aug 10 23:37:06 dist-test-slave-04bb krb5kdc[8011](info): commencing operation
krb5kdc: starting...
W20260810 23:37:07.906746 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.326s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:08 dist-test-slave-04bb krb5kdc[8011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:37:08.231973 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:37323
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:37323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:08.573908 8030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:08.574229 8030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:08.574401 8030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:08.582168 8030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:08.582293 8030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:08.582379 8030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:08.582479 8030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:08.582564 8030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:08.592363 8030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:37323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:37323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:08.593698 8030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:08.594998 8030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:08.607129 8037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:08.607081 8041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:08.607177 8038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:08.608029 8030 server_base.cc:1034] running on GCE node
I20260810 23:37:08.608906 8030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:08.610270 8030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:08.611506 8030 hybrid_clock.cc:648] HybridClock initialized: now 1786405028611470 us; error 48 us; skew 500 ppm
I20260810 23:37:08.611877 8030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:08 dist-test-slave-04bb krb5kdc[8011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405028, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:08.614995 8030 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:37:08.616408 8030 webserver.cc:460] Webserver started at http://127.9.48.254:36699/ using document root <none> and password file <none>
I20260810 23:37:08.617147 8030 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:08.617307 8030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:08.617621 8030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:08.620636 8030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "5773f0f2fd7448518ae2270a10477e40"
format_stamp: "Formatted at 2026-08-10 23:37:08 on dist-test-slave-04bb"
server_key: "4c80b3c8b97e4443512b69cb0ed13552"
server_key_iv: "28337068cd9b74ed3432a4daae257e72"
server_key_version: "encryptionkey@0"
I20260810 23:37:08.621409 8030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "5773f0f2fd7448518ae2270a10477e40"
format_stamp: "Formatted at 2026-08-10 23:37:08 on dist-test-slave-04bb"
server_key: "4c80b3c8b97e4443512b69cb0ed13552"
server_key_iv: "28337068cd9b74ed3432a4daae257e72"
server_key_version: "encryptionkey@0"
I20260810 23:37:08.626708 8030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:37:08.630168 8051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:08.631438 8030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:37:08.631625 8030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "5773f0f2fd7448518ae2270a10477e40"
format_stamp: "Formatted at 2026-08-10 23:37:08 on dist-test-slave-04bb"
server_key: "4c80b3c8b97e4443512b69cb0ed13552"
server_key_iv: "28337068cd9b74ed3432a4daae257e72"
server_key_version: "encryptionkey@0"
I20260810 23:37:08.631829 8030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:08.644179 8030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:08.648653 8030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:08.648942 8030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:08.668013 8030 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:37323
I20260810 23:37:08.668045 8128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:37323 every 8 connection(s)
I20260810 23:37:08.669488 8030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:08.673439 8132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:08.674008 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 8030
I20260810 23:37:08.674188 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:08.674677 9411 external_mini_cluster.cc:1428] Setting key 66aa99e293546e697b0143e124fb1f78
I20260810 23:37:08.688104 8132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40: Bootstrap starting.
Aug 10 23:37:08 dist-test-slave-04bb krb5kdc[8011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:37:08.691303 8132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:08.692797 8132 log.cc:824] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:08.696597 8132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40: No bootstrap required, opened a new log
I20260810 23:37:08.700964 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:08.676569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.9.48.254:37323)
0810 23:37:08.677471 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:08.677485 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:08.677538 (+ 53us) server_negotiation.cc:407] Connection header received
0810 23:37:08.680219 (+ 2681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:08.680249 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:08.680806 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:08.681251 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:08.682703 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:08.684836 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:08.686129 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:08.687171 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:08.687353 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:08.691721 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:08.691736 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:08.691757 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:08.691822 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:08.696102 (+ 4280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:08.696954 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:08.696963 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:08.696970 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:08.697226 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:08.697675 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:08.697680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:08.697683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:08.698356 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:08.698691 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:08.699091 (+ 400us) server_negotiation.cc:299] Negotiation successful
0810 23:37:08.699557 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":329,"threads_started":1}
I20260810 23:37:08.706362 8132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5773f0f2fd7448518ae2270a10477e40" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 37323 } }
I20260810 23:37:08.706673 8132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:08.706779 8132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5773f0f2fd7448518ae2270a10477e40, State: Initialized, Role: FOLLOWER
I20260810 23:37:08.707275 8132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5773f0f2fd7448518ae2270a10477e40" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 37323 } }
I20260810 23:37:08.707476 8132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:08.707583 8132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:08.707769 8132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:08.710227 8132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5773f0f2fd7448518ae2270a10477e40" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 37323 } }
I20260810 23:37:08.710704 8132 leader_election.cc:302] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5773f0f2fd7448518ae2270a10477e40; no voters:
I20260810 23:37:08.711340 8132 leader_election.cc:288] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:08.711515 8139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:08.712289 8139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [term 1 LEADER]: Becoming Leader. State: Replica: 5773f0f2fd7448518ae2270a10477e40, State: Running, Role: LEADER
I20260810 23:37:08.712890 8139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5773f0f2fd7448518ae2270a10477e40" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 37323 } }
I20260810 23:37:08.713462 8132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:08.716136 8141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5773f0f2fd7448518ae2270a10477e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5773f0f2fd7448518ae2270a10477e40" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 37323 } } }
I20260810 23:37:08.716648 8141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:08.716207 8143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5773f0f2fd7448518ae2270a10477e40. Latest consensus state: current_term: 1 leader_uuid: "5773f0f2fd7448518ae2270a10477e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5773f0f2fd7448518ae2270a10477e40" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 37323 } } }
I20260810 23:37:08.717052 8143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:08.718072 8154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:08.725322 8154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:08.739202 8154 catalog_manager.cc:1347] Generated new cluster ID: 4a869f8e19c24cb3b82092f9b09a6c7d
I20260810 23:37:08.739348 8154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:08.750586 8154 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:08.752419 8154 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:08.760231 8154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40: Generated new TSK 0
I20260810 23:37:08.761098 8154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.48.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
Entry for principal kudu/127.9.48.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab.
I20260810 23:37:08.856494 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:37323
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:09.212633 8177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:09.212901 8177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:09.213061 8177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:09.220360 8177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:09.220458 8177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:09.220686 8177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:37:09.230296 8177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.193.keytab
--principal=kudu/127.9.48.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:37323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:09.231547 8177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:09.232888 8177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:09.246038 8186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:09.246147 8187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:09.246170 8189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:09.247352 8177 server_base.cc:1034] running on GCE node
I20260810 23:37:09.247936 8177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:09.248955 8177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:09.250182 8177 hybrid_clock.cc:648] HybridClock initialized: now 1786405029250138 us; error 42 us; skew 500 ppm
I20260810 23:37:09.250494 8177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:09 dist-test-slave-04bb krb5kdc[8011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405029, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:09.253700 8177 init.cc:377] Logged in from keytab as kudu/127.9.48.193@KRBTEST.COM (short username kudu)
I20260810 23:37:09.255259 8177 webserver.cc:460] Webserver started at http://127.9.48.193:33145/ using document root <none> and password file <none>
I20260810 23:37:09.255895 8177 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:09.255997 8177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:09.256256 8177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:09.259179 8177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "650e5633aed84c71903b53bd7b0bbc99"
format_stamp: "Formatted at 2026-08-10 23:37:09 on dist-test-slave-04bb"
server_key: "dc3481774817996c69ed847ec8750827"
server_key_iv: "df9be487b06e9895dddaa9810d9c272b"
server_key_version: "encryptionkey@0"
I20260810 23:37:09.260032 8177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "650e5633aed84c71903b53bd7b0bbc99"
format_stamp: "Formatted at 2026-08-10 23:37:09 on dist-test-slave-04bb"
server_key: "dc3481774817996c69ed847ec8750827"
server_key_iv: "df9be487b06e9895dddaa9810d9c272b"
server_key_version: "encryptionkey@0"
I20260810 23:37:09.265362 8177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:37:09.268713 8202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:09.270008 8177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:37:09.270146 8177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "650e5633aed84c71903b53bd7b0bbc99"
format_stamp: "Formatted at 2026-08-10 23:37:09 on dist-test-slave-04bb"
server_key: "dc3481774817996c69ed847ec8750827"
server_key_iv: "df9be487b06e9895dddaa9810d9c272b"
server_key_version: "encryptionkey@0"
I20260810 23:37:09.270311 8177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:09.298821 8177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:09.302992 8177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:09.303242 8177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:09.304250 8177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:37:09.305594 8177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:37:09.305667 8177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:09.305753 8177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:37:09.305799 8177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:09.340844 8177 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:41835
I20260810 23:37:09.340937 8373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:41835 every 8 connection(s)
I20260810 23:37:09.342152 8177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:37:09.350558 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 8177
I20260810 23:37:09.350749 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:37:09.351284 9411 external_mini_cluster.cc:1428] Setting key f61eab5d623db34643c7ae54e25f220d
Aug 10 23:37:09 dist-test-slave-04bb krb5kdc[8011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405029, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.193@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:37:09.363927 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:09.344746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:42151 (local address 127.9.48.254:37323)
0810 23:37:09.345222 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:09.345226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:09.346138 (+ 912us) server_negotiation.cc:407] Connection header received
0810 23:37:09.348030 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:09.348036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:09.348168 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:09.348300 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:09.349880 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:09.350914 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:09.352882 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:09.353468 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:09.353602 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:09.357108 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:09.357117 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:09.357125 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:09.357171 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:09.360724 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:09.361438 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:09.361446 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:09.361452 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:09.361603 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:09.362023 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:09.362030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:09.362036 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:09.362406 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:09.362563 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:09.363194 (+ 631us) server_negotiation.cc:299] Negotiation successful
0810 23:37:09.363445 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":267,"threads_started":1}
I20260810 23:37:09.364497 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:09.345263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:37323 (local address 127.9.48.193:42151)
0810 23:37:09.345982 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:09.346020 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:09.347727 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:09.348586 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:09.348597 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:09.349127 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:09.349711 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:09.349727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:09.351087 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:09.351092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:09.352740 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:09.352754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:09.353684 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:09.353689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:09.353900 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:09.354581 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:37:09.354602 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:09.356866 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:09.360869 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:09.360876 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:09.360887 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:09.361309 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:09.361719 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:09.361722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:09.361725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:09.361906 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:09.362535 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:09.362540 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:09.362814 (+ 274us) client_negotiation.cc:769] Sending connection context
0810 23:37:09.363025 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 23:37:09.363390 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":285,"threads_started":1}
I20260810 23:37:09.366086 8374 heartbeater.cc:347] Connected to a master server at 127.9.48.254:37323
I20260810 23:37:09.366412 8374 heartbeater.cc:464] Registering TS with master...
I20260810 23:37:09.367255 8374 heartbeater.cc:511] Master 127.9.48.254:37323 requested a full tablet report, sending...
I20260810 23:37:09.369787 8073 ts_manager.cc:194] Registered new tserver with Master: 650e5633aed84c71903b53bd7b0bbc99 (127.9.48.193:41835)
I20260810 23:37:09.371924 8073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.193@KRBTEST.COM'} at 127.9.48.193:42151
WARNING: no policy specified for kudu/127.9.48.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
Entry for principal kudu/127.9.48.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab.
I20260810 23:37:09.423808 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:37323
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:09.782009 8383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:09.782284 8383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:09.782377 8383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:09.789485 8383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:09.789587 8383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:09.789696 8383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:37:09.799082 8383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.194.keytab
--principal=kudu/127.9.48.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:37323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:09.800267 8383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:09.801555 8383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:09.814913 8395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:09.814881 8392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:09.815390 8393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:09.816596 8383 server_base.cc:1034] running on GCE node
I20260810 23:37:09.817251 8383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:09.818202 8383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:09.819439 8383 hybrid_clock.cc:648] HybridClock initialized: now 1786405029819385 us; error 48 us; skew 500 ppm
I20260810 23:37:09.819759 8383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:09 dist-test-slave-04bb krb5kdc[8011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405029, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:09.823016 8383 init.cc:377] Logged in from keytab as kudu/127.9.48.194@KRBTEST.COM (short username kudu)
I20260810 23:37:09.824414 8383 webserver.cc:460] Webserver started at http://127.9.48.194:35491/ using document root <none> and password file <none>
I20260810 23:37:09.825129 8383 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:09.825233 8383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:09.825506 8383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:09.828451 8383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "49db64d1e9d54527af0c7279ac3d1377"
format_stamp: "Formatted at 2026-08-10 23:37:09 on dist-test-slave-04bb"
server_key: "a638fe02b3579a76842dd5b9f49ef0f0"
server_key_iv: "4771feba98a11850ddd85068d47eea19"
server_key_version: "encryptionkey@0"
I20260810 23:37:09.829243 8383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "49db64d1e9d54527af0c7279ac3d1377"
format_stamp: "Formatted at 2026-08-10 23:37:09 on dist-test-slave-04bb"
server_key: "a638fe02b3579a76842dd5b9f49ef0f0"
server_key_iv: "4771feba98a11850ddd85068d47eea19"
server_key_version: "encryptionkey@0"
I20260810 23:37:09.834905 8383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:37:09.838554 8403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:09.839886 8383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:37:09.840013 8383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "49db64d1e9d54527af0c7279ac3d1377"
format_stamp: "Formatted at 2026-08-10 23:37:09 on dist-test-slave-04bb"
server_key: "a638fe02b3579a76842dd5b9f49ef0f0"
server_key_iv: "4771feba98a11850ddd85068d47eea19"
server_key_version: "encryptionkey@0"
I20260810 23:37:09.840178 8383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:09.853775 8383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:09.859055 8383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:09.859390 8383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:09.860378 8383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:37:09.861835 8383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:37:09.861953 8383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:09.862114 8383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:37:09.862201 8383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:09.895900 8383 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:35787
I20260810 23:37:09.895951 8571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:35787 every 8 connection(s)
I20260810 23:37:09.897373 8383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:37:09.897735 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 8383
I20260810 23:37:09.897951 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:37:09.898603 9411 external_mini_cluster.cc:1428] Setting key 8c12d428997db05cae07ff93deb4dada
Aug 10 23:37:09 dist-test-slave-04bb krb5kdc[8011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405029, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.194@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:37:09.917910 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:09.900246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:40237 (local address 127.9.48.254:37323)
0810 23:37:09.900905 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:09.900911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:09.901956 (+ 1045us) server_negotiation.cc:407] Connection header received
0810 23:37:09.903775 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:09.903780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:09.903886 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:09.904009 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:09.905606 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:09.906368 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:09.907911 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:09.908367 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:09.908508 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:09.911958 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:09.911963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:09.911966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:09.911999 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:09.914820 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:09.915552 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:09.915557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:09.915561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:09.915695 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:09.916144 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:09.916149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:09.916152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:09.916439 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:09.916618 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:09.917304 (+ 686us) server_negotiation.cc:299] Negotiation successful
0810 23:37:09.917526 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":459,"threads_started":1}
I20260810 23:37:09.918848 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:09.900989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:37323 (local address 127.9.48.194:40237)
0810 23:37:09.901802 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:09.901849 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:09.903511 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:09.904256 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:09.904265 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:09.904871 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:09.905434 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:09.905451 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:09.906532 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:09.906536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:09.907739 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:09.907751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:09.908632 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:09.908640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:09.908872 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:09.909577 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:37:09.909597 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:09.911723 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:09.914968 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:09.914978 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:09.914995 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:09.915422 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:09.915818 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:09.915823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:09.915827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:09.916021 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:09.916592 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:09.916600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:09.916944 (+ 344us) client_negotiation.cc:769] Sending connection context
0810 23:37:09.917199 (+ 255us) client_negotiation.cc:240] Negotiation successful
0810 23:37:09.917724 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":355,"threads_started":1}
I20260810 23:37:09.920342 8572 heartbeater.cc:347] Connected to a master server at 127.9.48.254:37323
I20260810 23:37:09.920686 8572 heartbeater.cc:464] Registering TS with master...
I20260810 23:37:09.921432 8572 heartbeater.cc:511] Master 127.9.48.254:37323 requested a full tablet report, sending...
I20260810 23:37:09.922926 8073 ts_manager.cc:194] Registered new tserver with Master: 49db64d1e9d54527af0c7279ac3d1377 (127.9.48.194:35787)
I20260810 23:37:09.923738 8073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.194@KRBTEST.COM'} at 127.9.48.194:40237
WARNING: no policy specified for kudu/127.9.48.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
Entry for principal kudu/127.9.48.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab.
I20260810 23:37:09.973690 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:37323
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:10.331291 8583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:10.331573 8583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:10.331672 8583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:10.339282 8583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:10.339419 8583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:10.339546 8583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:37:10.352133 8583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.195.keytab
--principal=kudu/127.9.48.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:37323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:10.353423 8583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:10.354750 8583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:10.368671 8588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:10.368758 8592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:10.368721 8589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:10.369714 8583 server_base.cc:1034] running on GCE node
I20260810 23:37:10.370357 8583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:10.371352 8583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:10.372660 8583 hybrid_clock.cc:648] HybridClock initialized: now 1786405030372589 us; error 51 us; skew 500 ppm
I20260810 23:37:10.373068 8583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:37:10.375168 8374 heartbeater.cc:503] Master 127.9.48.254:37323 was elected leader, sending a full tablet report...
Aug 10 23:37:10 dist-test-slave-04bb krb5kdc[8011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405030, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:10.376868 8583 init.cc:377] Logged in from keytab as kudu/127.9.48.195@KRBTEST.COM (short username kudu)
I20260810 23:37:10.378479 8583 webserver.cc:460] Webserver started at http://127.9.48.195:45327/ using document root <none> and password file <none>
I20260810 23:37:10.379155 8583 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:10.379271 8583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:10.379532 8583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:10.382532 8583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "3a40af47a13a4a6ba69cb61cbf1908b4"
format_stamp: "Formatted at 2026-08-10 23:37:10 on dist-test-slave-04bb"
server_key: "f4291b00a27fa486fbcd5b7a40f79a8a"
server_key_iv: "cf83f0780630d7c339d885bb26b3749b"
server_key_version: "encryptionkey@0"
I20260810 23:37:10.383307 8583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "3a40af47a13a4a6ba69cb61cbf1908b4"
format_stamp: "Formatted at 2026-08-10 23:37:10 on dist-test-slave-04bb"
server_key: "f4291b00a27fa486fbcd5b7a40f79a8a"
server_key_iv: "cf83f0780630d7c339d885bb26b3749b"
server_key_version: "encryptionkey@0"
I20260810 23:37:10.388695 8583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 23:37:10.392395 8606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:10.393814 8583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:37:10.393952 8583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "3a40af47a13a4a6ba69cb61cbf1908b4"
format_stamp: "Formatted at 2026-08-10 23:37:10 on dist-test-slave-04bb"
server_key: "f4291b00a27fa486fbcd5b7a40f79a8a"
server_key_iv: "cf83f0780630d7c339d885bb26b3749b"
server_key_version: "encryptionkey@0"
I20260810 23:37:10.394116 8583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:10.421054 8583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:10.425698 8583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:10.425958 8583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:10.427120 8583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:37:10.428658 8583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:37:10.428740 8583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:10.428894 8583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:37:10.428962 8583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:10.465440 8583 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:45681
I20260810 23:37:10.465593 8787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:45681 every 8 connection(s)
I20260810 23:37:10.466909 8583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:37:10.476387 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 8583
I20260810 23:37:10.476615 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:37:10.477115 9411 external_mini_cluster.cc:1428] Setting key de03312a88558eacd1e771506addb0a0
Aug 10 23:37:10 dist-test-slave-04bb krb5kdc[8011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405030, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.195@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:37:10.489346 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:10.469822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:37115 (local address 127.9.48.254:37323)
0810 23:37:10.470386 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:10.470391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:10.471232 (+ 841us) server_negotiation.cc:407] Connection header received
0810 23:37:10.473346 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:10.473352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:10.473498 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:10.473651 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:10.475348 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.476244 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.478380 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.479089 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.479247 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:10.483398 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:10.483403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:10.483407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:10.483441 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:10.486422 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:10.487129 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:10.487133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:10.487135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:10.487289 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:10.487692 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:10.487695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:10.487698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:10.487990 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:10.488119 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:10.488853 (+ 734us) server_negotiation.cc:299] Negotiation successful
0810 23:37:10.489024 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":302,"threads_started":1}
I20260810 23:37:10.490257 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:10.470257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:37323 (local address 127.9.48.195:37115)
0810 23:37:10.471021 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:10.471065 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:10.473050 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:10.473915 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:10.473924 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:10.474464 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:10.475157 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:10.475176 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.476410 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.476421 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:10.478173 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:10.478189 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.479387 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.479398 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:10.479731 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:10.480693 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:37:10.480717 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:10.483119 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:10.486559 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:10.486566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:10.486578 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:10.487011 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:10.487391 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:10.487394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:10.487397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:10.487578 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:10.488107 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:10.488113 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:10.488417 (+ 304us) client_negotiation.cc:769] Sending connection context
0810 23:37:10.488726 (+ 309us) client_negotiation.cc:240] Negotiation successful
0810 23:37:10.489104 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":322,"threads_started":1}
I20260810 23:37:10.492038 8788 heartbeater.cc:347] Connected to a master server at 127.9.48.254:37323
I20260810 23:37:10.492484 8788 heartbeater.cc:464] Registering TS with master...
I20260810 23:37:10.493368 8788 heartbeater.cc:511] Master 127.9.48.254:37323 requested a full tablet report, sending...
I20260810 23:37:10.494827 8072 ts_manager.cc:194] Registered new tserver with Master: 3a40af47a13a4a6ba69cb61cbf1908b4 (127.9.48.195:45681)
I20260810 23:37:10.495760 8072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.48.195@KRBTEST.COM'} at 127.9.48.195:37115
I20260810 23:37:10.505587 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:37:10.522889 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:10.509860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.9.48.254:37323)
0810 23:37:10.510053 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:10.510058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:10.510611 (+ 553us) server_negotiation.cc:407] Connection header received
0810 23:37:10.510728 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:10.510732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:10.510798 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:10.510886 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:10.512356 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.513213 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.514722 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.515145 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.515251 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:10.517742 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:10.517747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:10.517750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:10.517781 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:10.520410 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:10.521127 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:10.521133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:10.521136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:10.521271 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:10.521597 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:10.521600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:10.521602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:10.521951 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:10.522152 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:10.522463 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 23:37:10.522642 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:37:10.527181 8072 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:37:10.534426 8072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:37:10.557410 8806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:10.546945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:41835 (local address 127.0.0.1:45052)
0810 23:37:10.547742 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:10.547796 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:10.547981 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:10.549651 (+ 1670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:10.549661 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:10.549703 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:10.550235 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:10.550254 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.552314 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.552319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:10.554741 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:10.554754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.556286 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.556291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:10.556479 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:10.556531 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 23:37:10.556718 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 23:37:10.556924 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":106,"thread_start_us":384,"threads_started":1}
I20260810 23:37:10.558219 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:10.547030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:35787 (local address 127.0.0.1:45844)
0810 23:37:10.548484 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:10.548503 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:10.548647 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:10.549723 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:10.549727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:10.549772 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:10.550317 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:10.550326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.552549 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.552557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:10.555450 (+ 2893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:10.555465 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.557372 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.557379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:10.557567 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:10.557602 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:37:10.557678 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 23:37:10.557811 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1314,"mutex_wait_us":188,"spinlock_wait_cycles":896,"thread_start_us":1084,"threads_started":1}
I20260810 23:37:10.558259 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:10.547293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.9.48.193:41835)
0810 23:37:10.548693 (+ 1400us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:10.548704 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:10.548737 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:37:10.548848 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:10.548858 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:10.549144 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:10.549346 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:10.550481 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.552170 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.554895 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.556058 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.556210 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:10.556470 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:10.557876 (+ 1406us) server_negotiation.cc:299] Negotiation successful
0810 23:37:10.558020 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":1024,"threads_started":1}
I20260810 23:37:10.559333 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:10.547898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.9.48.194:35787)
0810 23:37:10.549144 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:10.549153 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:10.549183 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:37:10.549288 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:10.549297 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:10.549515 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:10.550506 (+ 991us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:10.550888 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.552393 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.555613 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.557099 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.558591 (+ 1492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:10.558781 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:10.558853 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 23:37:10.559000 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1080,"thread_start_us":948,"threads_started":1}
I20260810 23:37:10.560999 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:10.548638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:45681 (local address 127.0.0.1:33292)
0810 23:37:10.549426 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:10.549446 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:10.549570 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:10.550822 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:10.550827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:10.550854 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:10.551228 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:10.551238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.552798 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.552808 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:10.555451 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:10.555465 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.559417 (+ 3952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.559425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:10.560327 (+ 902us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:10.560362 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:37:10.560466 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 23:37:10.560614 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":243,"thread_start_us":939,"threads_started":1}
I20260810 23:37:10.561323 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:10.549317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.9.48.195:45681)
0810 23:37:10.550155 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:10.550163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:10.550194 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:37:10.550283 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:10.550290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:10.550517 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:10.550829 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:10.551406 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.552601 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.556618 (+ 4017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.558230 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.559676 (+ 1446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:10.559899 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:10.560554 (+ 655us) server_negotiation.cc:299] Negotiation successful
0810 23:37:10.560730 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":1264,"threads_started":1}
I20260810 23:37:10.562842 8471 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8f42884a8043379444ca716303560a (DEFAULT_TABLE table=test-table [id=880b6d92acf24fefb954f1d55b776687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:37:10.564185 8471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f8f42884a8043379444ca716303560a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:10.563719 8267 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8f42884a8043379444ca716303560a (DEFAULT_TABLE table=test-table [id=880b6d92acf24fefb954f1d55b776687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:37:10.565259 8267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f8f42884a8043379444ca716303560a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:10.564706 8682 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8f42884a8043379444ca716303560a (DEFAULT_TABLE table=test-table [id=880b6d92acf24fefb954f1d55b776687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:37:10.566121 8682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f8f42884a8043379444ca716303560a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:10.576099 8821 tablet_bootstrap.cc:492] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377: Bootstrap starting.
I20260810 23:37:10.579864 8822 tablet_bootstrap.cc:492] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99: Bootstrap starting.
I20260810 23:37:10.581183 8821 tablet_bootstrap.cc:654] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:10.581679 8824 tablet_bootstrap.cc:492] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4: Bootstrap starting.
I20260810 23:37:10.582718 8821 log.cc:824] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:10.583613 8822 tablet_bootstrap.cc:654] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:10.584830 8822 log.cc:824] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:10.584889 8824 tablet_bootstrap.cc:654] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:10.585978 8821 tablet_bootstrap.cc:492] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377: No bootstrap required, opened a new log
I20260810 23:37:10.586056 8824 log.cc:824] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:10.586297 8821 ts_tablet_manager.cc:1397] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260810 23:37:10.587929 8822 tablet_bootstrap.cc:492] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99: No bootstrap required, opened a new log
I20260810 23:37:10.588195 8822 ts_tablet_manager.cc:1397] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 23:37:10.588796 8824 tablet_bootstrap.cc:492] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4: No bootstrap required, opened a new log
I20260810 23:37:10.589036 8824 ts_tablet_manager.cc:1397] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260810 23:37:10.595963 8824 raft_consensus.cc:348] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "650e5633aed84c71903b53bd7b0bbc99" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 41835 } } peers { permanent_uuid: "49db64d1e9d54527af0c7279ac3d1377" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35787 } } peers { permanent_uuid: "3a40af47a13a4a6ba69cb61cbf1908b4" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45681 } }
I20260810 23:37:10.596347 8824 raft_consensus.cc:374] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:10.596457 8824 raft_consensus.cc:729] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a40af47a13a4a6ba69cb61cbf1908b4, State: Initialized, Role: FOLLOWER
I20260810 23:37:10.597289 8824 consensus_queue.cc:260] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "650e5633aed84c71903b53bd7b0bbc99" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 41835 } } peers { permanent_uuid: "49db64d1e9d54527af0c7279ac3d1377" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35787 } } peers { permanent_uuid: "3a40af47a13a4a6ba69cb61cbf1908b4" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45681 } }
I20260810 23:37:10.597692 8821 raft_consensus.cc:348] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "650e5633aed84c71903b53bd7b0bbc99" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 41835 } } peers { permanent_uuid: "49db64d1e9d54527af0c7279ac3d1377" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35787 } } peers { permanent_uuid: "3a40af47a13a4a6ba69cb61cbf1908b4" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45681 } }
I20260810 23:37:10.598141 8821 raft_consensus.cc:374] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:10.598240 8821 raft_consensus.cc:729] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49db64d1e9d54527af0c7279ac3d1377, State: Initialized, Role: FOLLOWER
I20260810 23:37:10.598845 8788 heartbeater.cc:503] Master 127.9.48.254:37323 was elected leader, sending a full tablet report...
I20260810 23:37:10.598769 8822 raft_consensus.cc:348] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "650e5633aed84c71903b53bd7b0bbc99" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 41835 } } peers { permanent_uuid: "49db64d1e9d54527af0c7279ac3d1377" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35787 } } peers { permanent_uuid: "3a40af47a13a4a6ba69cb61cbf1908b4" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45681 } }
I20260810 23:37:10.598963 8821 consensus_queue.cc:260] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "650e5633aed84c71903b53bd7b0bbc99" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 41835 } } peers { permanent_uuid: "49db64d1e9d54527af0c7279ac3d1377" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35787 } } peers { permanent_uuid: "3a40af47a13a4a6ba69cb61cbf1908b4" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45681 } }
I20260810 23:37:10.599169 8822 raft_consensus.cc:374] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:10.599268 8822 raft_consensus.cc:729] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 650e5633aed84c71903b53bd7b0bbc99, State: Initialized, Role: FOLLOWER
I20260810 23:37:10.599329 8824 ts_tablet_manager.cc:1428] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4: Time spent starting tablet: real 0.010s user 0.002s sys 0.009s
I20260810 23:37:10.599890 8822 consensus_queue.cc:260] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "650e5633aed84c71903b53bd7b0bbc99" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 41835 } } peers { permanent_uuid: "49db64d1e9d54527af0c7279ac3d1377" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35787 } } peers { permanent_uuid: "3a40af47a13a4a6ba69cb61cbf1908b4" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45681 } }
I20260810 23:37:10.602277 8822 ts_tablet_manager.cc:1428] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260810 23:37:10.602262 8821 ts_tablet_manager.cc:1428] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260810 23:37:10.602810 8572 heartbeater.cc:503] Master 127.9.48.254:37323 was elected leader, sending a full tablet report...
W20260810 23:37:10.649596 8574 tablet.cc:2369] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:37:10.718856 8789 tablet.cc:2369] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:37:10.735507 8834 raft_consensus.cc:482] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:37:10.735826 8834 raft_consensus.cc:504] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "650e5633aed84c71903b53bd7b0bbc99" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 41835 } } peers { permanent_uuid: "49db64d1e9d54527af0c7279ac3d1377" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35787 } } peers { permanent_uuid: "3a40af47a13a4a6ba69cb61cbf1908b4" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45681 } }
I20260810 23:37:10.737691 8834 leader_election.cc:288] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 650e5633aed84c71903b53bd7b0bbc99 (127.9.48.193:41835), 3a40af47a13a4a6ba69cb61cbf1908b4 (127.9.48.195:45681)
I20260810 23:37:10.745522 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:10.738427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:51059 (local address 127.9.48.195:45681)
0810 23:37:10.738618 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:10.738626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:10.739466 (+ 840us) server_negotiation.cc:407] Connection header received
0810 23:37:10.740015 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:10.740024 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:10.740102 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:10.740167 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:10.740983 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.741760 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.743583 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.744450 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.744691 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:10.744789 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:10.745128 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 23:37:10.745260 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:37:10.745679 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:10.738072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:45681 (local address 127.9.48.194:51059)
0810 23:37:10.739328 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:10.739344 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:10.739806 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:10.740383 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:10.740386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:10.740414 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:10.740811 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:10.740830 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.741906 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.741912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:10.743415 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:10.743428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.744777 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.744785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:10.744953 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:10.744979 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:37:10.745101 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 23:37:10.745232 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"mutex_wait_us":62,"thread_start_us":747,"threads_started":1}
I20260810 23:37:10.745731 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:10.738309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:35645 (local address 127.9.48.193:41835)
0810 23:37:10.738523 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:10.738529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:10.739418 (+ 889us) server_negotiation.cc:407] Connection header received
0810 23:37:10.739598 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:10.739603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:10.739674 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:10.739815 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:10.740451 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.741472 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.743598 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.744645 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.744841 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:10.744935 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:10.745303 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 23:37:10.745439 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:37:10.745900 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:10.738072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:41835 (local address 127.9.48.194:35645)
0810 23:37:10.739227 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:10.739250 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:10.739427 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:10.739853 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:10.739861 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:10.739891 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:10.740277 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:10.740292 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.741636 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.741645 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:10.743417 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:10.743432 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.744893 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.744905 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:10.745119 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:10.745152 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:37:10.745273 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 23:37:10.745422 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":6,"thread_start_us":626,"threads_started":1}
I20260810 23:37:10.746968 8301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8f42884a8043379444ca716303560a" candidate_uuid: "49db64d1e9d54527af0c7279ac3d1377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "650e5633aed84c71903b53bd7b0bbc99" is_pre_election: true
I20260810 23:37:10.747025 8719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8f42884a8043379444ca716303560a" candidate_uuid: "49db64d1e9d54527af0c7279ac3d1377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a40af47a13a4a6ba69cb61cbf1908b4" is_pre_election: true
I20260810 23:37:10.747496 8301 raft_consensus.cc:2454] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49db64d1e9d54527af0c7279ac3d1377 in term 0.
I20260810 23:37:10.747540 8719 raft_consensus.cc:2454] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49db64d1e9d54527af0c7279ac3d1377 in term 0.
I20260810 23:37:10.748271 8404 leader_election.cc:302] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49db64d1e9d54527af0c7279ac3d1377, 650e5633aed84c71903b53bd7b0bbc99; no voters:
I20260810 23:37:10.748719 8834 raft_consensus.cc:2781] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:37:10.748876 8834 raft_consensus.cc:482] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:37:10.748997 8834 raft_consensus.cc:3037] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:10.751235 8834 raft_consensus.cc:504] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "650e5633aed84c71903b53bd7b0bbc99" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 41835 } } peers { permanent_uuid: "49db64d1e9d54527af0c7279ac3d1377" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35787 } } peers { permanent_uuid: "3a40af47a13a4a6ba69cb61cbf1908b4" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45681 } }
I20260810 23:37:10.751891 8834 leader_election.cc:288] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [CANDIDATE]: Term 1 election: Requested vote from peers 650e5633aed84c71903b53bd7b0bbc99 (127.9.48.193:41835), 3a40af47a13a4a6ba69cb61cbf1908b4 (127.9.48.195:45681)
I20260810 23:37:10.752372 8301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8f42884a8043379444ca716303560a" candidate_uuid: "49db64d1e9d54527af0c7279ac3d1377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "650e5633aed84c71903b53bd7b0bbc99"
I20260810 23:37:10.752483 8719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8f42884a8043379444ca716303560a" candidate_uuid: "49db64d1e9d54527af0c7279ac3d1377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a40af47a13a4a6ba69cb61cbf1908b4"
I20260810 23:37:10.752597 8301 raft_consensus.cc:3037] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:10.752781 8719 raft_consensus.cc:3037] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:10.754863 8301 raft_consensus.cc:2454] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49db64d1e9d54527af0c7279ac3d1377 in term 1.
I20260810 23:37:10.755275 8404 leader_election.cc:302] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49db64d1e9d54527af0c7279ac3d1377, 650e5633aed84c71903b53bd7b0bbc99; no voters:
I20260810 23:37:10.755281 8719 raft_consensus.cc:2454] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49db64d1e9d54527af0c7279ac3d1377 in term 1.
I20260810 23:37:10.755570 8834 raft_consensus.cc:2781] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:10.756037 8834 raft_consensus.cc:686] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [term 1 LEADER]: Becoming Leader. State: Replica: 49db64d1e9d54527af0c7279ac3d1377, State: Running, Role: LEADER
I20260810 23:37:10.756582 8834 consensus_queue.cc:237] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "650e5633aed84c71903b53bd7b0bbc99" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 41835 } } peers { permanent_uuid: "49db64d1e9d54527af0c7279ac3d1377" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35787 } } peers { permanent_uuid: "3a40af47a13a4a6ba69cb61cbf1908b4" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45681 } }
I20260810 23:37:10.761587 8071 catalog_manager.cc:5495] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49db64d1e9d54527af0c7279ac3d1377 (127.9.48.194). New cstate: current_term: 1 leader_uuid: "49db64d1e9d54527af0c7279ac3d1377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "650e5633aed84c71903b53bd7b0bbc99" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 41835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49db64d1e9d54527af0c7279ac3d1377" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 35787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a40af47a13a4a6ba69cb61cbf1908b4" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45681 } health_report { overall_health: UNKNOWN } } }
I20260810 23:37:10.799968 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:10.794453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.9.48.194:35787)
0810 23:37:10.794719 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:10.794724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:10.794741 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:37:10.794833 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:10.794840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:10.794918 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:10.795080 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:37:10.795491 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.796224 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.797704 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.798210 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.798347 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:10.798892 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:37:10.799384 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:37:10.799555 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:10.799619 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 23:37:10.799701 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 23:37:10.808785 8719 raft_consensus.cc:1261] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4 [term 1 FOLLOWER]: Refusing update from remote peer 49db64d1e9d54527af0c7279ac3d1377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:37:10.808792 8301 raft_consensus.cc:1261] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99 [term 1 FOLLOWER]: Refusing update from remote peer 49db64d1e9d54527af0c7279ac3d1377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:37:10.809911 8838 consensus_queue.cc:1035] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a40af47a13a4a6ba69cb61cbf1908b4" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 45681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:37:10.810320 8834 consensus_queue.cc:1035] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "650e5633aed84c71903b53bd7b0bbc99" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:37:10.824353 8844 mvcc.cc:205] Tried to move back new op lower bound from 7317115006180794368 to 7317115005982822400. Current Snapshot: MvccSnapshot[applied={T|T < 7317115006180794368}]
I20260810 23:37:10.828934 8843 mvcc.cc:205] Tried to move back new op lower bound from 7317115006180794368 to 7317115005982822400. Current Snapshot: MvccSnapshot[applied={T|T < 7317115006180794368}]
I20260810 23:37:10.847278 8071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:37:10.847640 8071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:37:10.851676 8071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5773f0f2fd7448518ae2270a10477e40: Sending DeleteTablet for 3 replicas of tablet 8f8f42884a8043379444ca716303560a
I20260810 23:37:10.852928 8267 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8f42884a8043379444ca716303560a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:37:10 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:45052
I20260810 23:37:10.853137 8682 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8f42884a8043379444ca716303560a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:37:10 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:33292
I20260810 23:37:10.853816 8471 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8f42884a8043379444ca716303560a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:37:10 UTC) from {username='kudu', principal='kudu/127.9.48.254@KRBTEST.COM'} at 127.0.0.1:45844
I20260810 23:37:10.853967 8857 tablet_replica.cc:330] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99: stopping tablet replica
I20260810 23:37:10.854511 8857 raft_consensus.cc:2229] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:37:10.855119 8857 raft_consensus.cc:2258] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:37:10.858515 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 8177
I20260810 23:37:10.867027 8857 ts_tablet_manager.cc:1905] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:37:10.869091 8861 tablet_replica.cc:330] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377: stopping tablet replica
I20260810 23:37:10.869570 8861 raft_consensus.cc:2229] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:37:10.870345 8861 raft_consensus.cc:2258] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:37:10.871182 8858 tablet_replica.cc:330] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4: stopping tablet replica
I20260810 23:37:10.871690 8858 raft_consensus.cc:2229] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:37:10.872303 8858 raft_consensus.cc:2258] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:37:10.872735 8861 ts_tablet_manager.cc:1905] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:37:10.879434 8857 ts_tablet_manager.cc:1918] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:37:10.879642 8857 log.cc:1196] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-0/wal/wals/8f8f42884a8043379444ca716303560a
I20260810 23:37:10.880295 8857 ts_tablet_manager.cc:1939] T 8f8f42884a8043379444ca716303560a P 650e5633aed84c71903b53bd7b0bbc99: Deleting consensus metadata
I20260810 23:37:10.880275 8861 ts_tablet_manager.cc:1918] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:37:10.880462 8861 log.cc:1196] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/8f8f42884a8043379444ca716303560a
I20260810 23:37:10.881184 8861 ts_tablet_manager.cc:1939] T 8f8f42884a8043379444ca716303560a P 49db64d1e9d54527af0c7279ac3d1377: Deleting consensus metadata
I20260810 23:37:10.882562 8053 catalog_manager.cc:4841] TS 650e5633aed84c71903b53bd7b0bbc99 (127.9.48.193:41835): tablet 8f8f42884a8043379444ca716303560a (table test-table [id=880b6d92acf24fefb954f1d55b776687]) successfully deleted
I20260810 23:37:10.883041 8056 catalog_manager.cc:4841] TS 49db64d1e9d54527af0c7279ac3d1377 (127.9.48.194:35787): tablet 8f8f42884a8043379444ca716303560a (table test-table [id=880b6d92acf24fefb954f1d55b776687]) successfully deleted
I20260810 23:37:10.886533 8858 ts_tablet_manager.cc:1905] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 23:37:10 dist-test-slave-04bb krb5kdc[8011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.193@KRBTEST.COM
I20260810 23:37:10.893036 8858 ts_tablet_manager.cc:1918] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:37:10.893217 8858 log.cc:1196] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/8f8f42884a8043379444ca716303560a
I20260810 23:37:10.893661 8858 ts_tablet_manager.cc:1939] T 8f8f42884a8043379444ca716303560a P 3a40af47a13a4a6ba69cb61cbf1908b4: Deleting consensus metadata
I20260810 23:37:10.896708 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:10.859682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.9.48.193:41835)
0810 23:37:10.859863 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:10.859870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:10.875429 (+ 15559us) server_negotiation.cc:407] Connection header received
0810 23:37:10.875644 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:10.875651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:10.875760 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:10.875888 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:10.878111 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.879474 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.882221 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:10.882934 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:10.883054 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:10.889013 (+ 5959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:10.889031 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:10.889049 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:10.889094 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:10.893151 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:10.894203 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:10.894215 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:10.894219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:10.894401 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:10.894883 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:10.894888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:10.894891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:10.895419 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:10.895737 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:10.896053 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 23:37:10.896311 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:37:10.900945 8056 catalog_manager.cc:4841] TS 3a40af47a13a4a6ba69cb61cbf1908b4 (127.9.48.195:45681): tablet 8f8f42884a8043379444ca716303560a (table test-table [id=880b6d92acf24fefb954f1d55b776687]) successfully deleted
I20260810 23:37:10.901535 8363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:10.992791 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 8177
I20260810 23:37:11.016862 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 8383
Aug 10 23:37:11 dist-test-slave-04bb krb5kdc[8011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.194@KRBTEST.COM
I20260810 23:37:11.034653 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:11.017715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.9.48.194:35787)
0810 23:37:11.017911 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:11.017919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:11.017942 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:37:11.018217 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:11.018221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:11.018292 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:11.018406 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:11.019687 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:11.020502 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:11.021851 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:11.022311 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:11.022465 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:11.027773 (+ 5308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:11.027789 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:11.027806 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:11.027864 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:11.031074 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:11.031973 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:11.031980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:11.031983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:11.032141 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:11.032615 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:11.032622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:11.032628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:11.033176 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:11.033520 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:11.033841 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 23:37:11.034130 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 23:37:11.035418 8564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:11.150015 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 8383
I20260810 23:37:11.178767 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 8583
Aug 10 23:37:11 dist-test-slave-04bb krb5kdc[8011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.195@KRBTEST.COM
I20260810 23:37:11.194589 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:11.179540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33298 (local address 127.9.48.195:45681)
0810 23:37:11.179770 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:11.179775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:11.179796 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:37:11.179956 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:11.179960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:11.180039 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:11.180122 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:11.181538 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:11.182261 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:11.183543 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:11.183934 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:11.184028 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:11.188804 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:11.188824 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:11.188852 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:11.188891 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:11.191890 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:11.192616 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:11.192624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:11.192626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:11.192778 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:11.193238 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:11.193242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:11.193244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:11.193589 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:11.193813 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:11.194115 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 23:37:11.194328 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:37:11.195403 8776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:11.285836 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 8583
I20260810 23:37:11.312080 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 8030
I20260810 23:37:11.313372 8123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:11.425500 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 8030
2026-08-10T23:37:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4897 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T23:37:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:11Z Disabled control of system clock
I20260810 23:37:11.527294 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:33967
--webserver_interface=127.9.48.254
--webserver_port=0
--builtin_ntp_servers=127.9.48.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:33967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:37:11.875627 8878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:11.875912 8878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:11.875996 8878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:11.883275 8878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:11.883384 8878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:11.883424 8878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:11.883476 8878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:11.883520 8878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:11.893093 8878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:33967
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:33967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:11.894850 8878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:11.896719 8878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:11.909133 8885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:11.909229 8889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:11.909263 8884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:11.910291 8878 server_base.cc:1034] running on GCE node
I20260810 23:37:11.911124 8878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:11.912703 8878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:11.913966 8878 hybrid_clock.cc:648] HybridClock initialized: now 1786405031913926 us; error 48 us; skew 500 ppm
I20260810 23:37:11.914389 8878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:37:11.916949 8878 webserver.cc:460] Webserver started at http://127.9.48.254:41087/ using document root <none> and password file <none>
I20260810 23:37:11.917678 8878 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:11.917836 8878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:11.918216 8878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:11.921433 8878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "c18fd87a0d4a486884ef47f6e6d3ab51"
format_stamp: "Formatted at 2026-08-10 23:37:11 on dist-test-slave-04bb"
server_key: "c0771b325b3e28ad175aced2a448cb38"
server_key_iv: "610fb863dfca3ab602da4f906003ea43"
server_key_version: "encryptionkey@0"
I20260810 23:37:11.922318 8878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "c18fd87a0d4a486884ef47f6e6d3ab51"
format_stamp: "Formatted at 2026-08-10 23:37:11 on dist-test-slave-04bb"
server_key: "c0771b325b3e28ad175aced2a448cb38"
server_key_iv: "610fb863dfca3ab602da4f906003ea43"
server_key_version: "encryptionkey@0"
I20260810 23:37:11.927680 8878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 23:37:11.931344 8895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:11.932821 8878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:37:11.933012 8878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "c18fd87a0d4a486884ef47f6e6d3ab51"
format_stamp: "Formatted at 2026-08-10 23:37:11 on dist-test-slave-04bb"
server_key: "c0771b325b3e28ad175aced2a448cb38"
server_key_iv: "610fb863dfca3ab602da4f906003ea43"
server_key_version: "encryptionkey@0"
I20260810 23:37:11.933250 8878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:11.953961 8878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:11.954995 8878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:11.955313 8878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:11.975698 8878 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:33967
I20260810 23:37:11.975754 8975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:33967 every 8 connection(s)
I20260810 23:37:11.977120 8878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:11.977572 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 8878
I20260810 23:37:11.977787 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:11.978545 9411 external_mini_cluster.cc:1428] Setting key ea5d3118711402873d70e4f88e62e112
I20260810 23:37:11.981690 8976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:11.993546 8985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:11.980928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.9.48.254:33967)
0810 23:37:11.982127 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:11.982140 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:11.982193 (+ 53us) server_negotiation.cc:407] Connection header received
0810 23:37:11.983945 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:11.983967 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:11.984473 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:11.984934 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:11.985554 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:11.987047 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:11.989032 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:11.989697 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:11.989836 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:11.991160 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:11.991170 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:11.991184 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:11.991789 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:11.991878 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:11.991899 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:11.991990 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:11.992247 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 23:37:11.992603 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":438,"threads_started":1}
I20260810 23:37:11.994330 8976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51: Bootstrap starting.
I20260810 23:37:11.997952 8976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:11.998979 8976 log.cc:824] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:12.001900 8976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51: No bootstrap required, opened a new log
I20260810 23:37:12.009824 8976 raft_consensus.cc:348] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18fd87a0d4a486884ef47f6e6d3ab51" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33967 } }
I20260810 23:37:12.010200 8976 raft_consensus.cc:374] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:12.010301 8976 raft_consensus.cc:729] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18fd87a0d4a486884ef47f6e6d3ab51, State: Initialized, Role: FOLLOWER
I20260810 23:37:12.010871 8976 consensus_queue.cc:260] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18fd87a0d4a486884ef47f6e6d3ab51" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33967 } }
I20260810 23:37:12.011107 8976 raft_consensus.cc:388] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:12.011272 8976 raft_consensus.cc:482] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:12.011466 8976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:12.013765 8976 raft_consensus.cc:504] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18fd87a0d4a486884ef47f6e6d3ab51" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33967 } }
I20260810 23:37:12.014283 8976 leader_election.cc:302] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c18fd87a0d4a486884ef47f6e6d3ab51; no voters:
I20260810 23:37:12.014983 8976 leader_election.cc:288] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:12.015116 8991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:12.015897 8991 raft_consensus.cc:686] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [term 1 LEADER]: Becoming Leader. State: Replica: c18fd87a0d4a486884ef47f6e6d3ab51, State: Running, Role: LEADER
I20260810 23:37:12.016458 8991 consensus_queue.cc:237] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18fd87a0d4a486884ef47f6e6d3ab51" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33967 } }
I20260810 23:37:12.017434 8976 sys_catalog.cc:564] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:12.019536 8993 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c18fd87a0d4a486884ef47f6e6d3ab51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18fd87a0d4a486884ef47f6e6d3ab51" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33967 } } }
I20260810 23:37:12.019903 8993 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:12.020382 8994 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c18fd87a0d4a486884ef47f6e6d3ab51. Latest consensus state: current_term: 1 leader_uuid: "c18fd87a0d4a486884ef47f6e6d3ab51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18fd87a0d4a486884ef47f6e6d3ab51" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 33967 } } }
I20260810 23:37:12.020658 8994 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:12.022432 9002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:12.029183 9002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:12.039754 9002 catalog_manager.cc:1347] Generated new cluster ID: 1fc7ad6c8876421885a8e0d472d3fd41
I20260810 23:37:12.039875 9002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:12.051836 9002 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:12.053531 9002 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:12.068929 9002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51: Generated new TSK 0
I20260810 23:37:12.069811 9002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:12.079792 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.193:0
--local_ip_for_outbound_sockets=127.9.48.193
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33967
--builtin_ntp_servers=127.9.48.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:37:12.433983 9020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:12.434299 9020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:12.434463 9020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:12.442001 9020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:12.442144 9020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:12.442309 9020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.193
I20260810 23:37:12.451390 9020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.48.193
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.193
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:12.452809 9020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:12.454591 9020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:12.468130 9028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:12.468051 9029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:12.468104 9031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:12.469272 9020 server_base.cc:1034] running on GCE node
I20260810 23:37:12.469954 9020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:12.470985 9020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:12.472263 9020 hybrid_clock.cc:648] HybridClock initialized: now 1786405032472229 us; error 70 us; skew 500 ppm
I20260810 23:37:12.472676 9020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:37:12.475318 9020 webserver.cc:460] Webserver started at http://127.9.48.193:41279/ using document root <none> and password file <none>
I20260810 23:37:12.475982 9020 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:12.476114 9020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:12.476420 9020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:12.479485 9020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/data/instance:
uuid: "7e8e4b1eb75d4d36925424942571c4c2"
format_stamp: "Formatted at 2026-08-10 23:37:12 on dist-test-slave-04bb"
server_key: "29226b5d79fa2d340d26775d16234680"
server_key_iv: "cab7ab48a1959eff95474725312ab2d4"
server_key_version: "encryptionkey@0"
I20260810 23:37:12.480252 9020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance:
uuid: "7e8e4b1eb75d4d36925424942571c4c2"
format_stamp: "Formatted at 2026-08-10 23:37:12 on dist-test-slave-04bb"
server_key: "29226b5d79fa2d340d26775d16234680"
server_key_iv: "cab7ab48a1959eff95474725312ab2d4"
server_key_version: "encryptionkey@0"
I20260810 23:37:12.485659 9020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:37:12.489356 9039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:12.490744 9020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:37:12.490928 9020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/wal
uuid: "7e8e4b1eb75d4d36925424942571c4c2"
format_stamp: "Formatted at 2026-08-10 23:37:12 on dist-test-slave-04bb"
server_key: "29226b5d79fa2d340d26775d16234680"
server_key_iv: "cab7ab48a1959eff95474725312ab2d4"
server_key_version: "encryptionkey@0"
I20260810 23:37:12.491154 9020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:12.512368 9020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:12.513216 9020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:12.513535 9020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:12.514508 9020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:37:12.515980 9020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:37:12.516104 9020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:12.516245 9020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:37:12.516340 9020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:12.551965 9020 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.193:42625
I20260810 23:37:12.552058 9209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.193:42625 every 8 connection(s)
I20260810 23:37:12.553318 9020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/data/info.pb
I20260810 23:37:12.557724 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 9020
I20260810 23:37:12.557927 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-0/wal/instance
I20260810 23:37:12.558396 9411 external_mini_cluster.cc:1428] Setting key 0308417753d0071e270c5d773c096caa
I20260810 23:37:12.561389 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.194:0
--local_ip_for_outbound_sockets=127.9.48.194
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33967
--builtin_ntp_servers=127.9.48.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:37:12.566864 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:12.556078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:34691 (local address 127.9.48.254:33967)
0810 23:37:12.556643 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:12.556650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:12.557720 (+ 1070us) server_negotiation.cc:407] Connection header received
0810 23:37:12.559962 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:12.559967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:12.560033 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:12.560181 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:12.561400 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:12.562307 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:12.563869 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:12.564294 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:12.564438 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:12.565939 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:12.565946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:12.565952 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:12.565977 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:12.566019 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:12.566032 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:12.566183 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:12.566422 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 23:37:12.566523 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":273,"threads_started":1}
I20260810 23:37:12.567979 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:12.556615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33967 (local address 127.9.48.193:34691)
0810 23:37:12.557550 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:12.557587 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:12.559657 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:12.560466 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:12.560480 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:12.560653 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:12.561227 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:12.561243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:12.562469 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:12.562475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:12.563710 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:12.563723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:12.564565 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:12.564575 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:12.564826 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:12.565585 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:37:12.565588 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:12.565639 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:37:12.565722 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:12.566183 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:12.566198 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:12.566209 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 23:37:12.566392 (+ 183us) client_negotiation.cc:240] Negotiation successful
0810 23:37:12.566813 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":289,"threads_started":1}
I20260810 23:37:12.569790 9210 heartbeater.cc:347] Connected to a master server at 127.9.48.254:33967
I20260810 23:37:12.570137 9210 heartbeater.cc:464] Registering TS with master...
I20260810 23:37:12.571059 9210 heartbeater.cc:511] Master 127.9.48.254:33967 requested a full tablet report, sending...
I20260810 23:37:12.573498 8921 ts_manager.cc:194] Registered new tserver with Master: 7e8e4b1eb75d4d36925424942571c4c2 (127.9.48.193:42625)
I20260810 23:37:12.575659 8921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.48.193:34691
W20260810 23:37:12.912699 9217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:12.913033 9217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:12.913195 9217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:12.920745 9217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:12.920871 9217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:12.921043 9217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.194
I20260810 23:37:12.930284 9217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.48.194
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.194
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:12.931643 9217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:12.933569 9217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:12.948817 9227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:12.948788 9228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:12.953133 9232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:12.954104 9217 server_base.cc:1034] running on GCE node
I20260810 23:37:12.954798 9217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:12.955883 9217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:12.957119 9217 hybrid_clock.cc:648] HybridClock initialized: now 1786405032957067 us; error 42 us; skew 500 ppm
I20260810 23:37:12.957463 9217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:37:12.960107 9217 webserver.cc:460] Webserver started at http://127.9.48.194:39915/ using document root <none> and password file <none>
I20260810 23:37:12.960893 9217 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:12.960984 9217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:12.961369 9217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:12.964385 9217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/data/instance:
uuid: "4a66cb6c53404d718ed33403179eb5c7"
format_stamp: "Formatted at 2026-08-10 23:37:12 on dist-test-slave-04bb"
server_key: "059afeda6f8d13ebc983965a11d5d56c"
server_key_iv: "a9ec3e71a627b14842ce71924ff94c20"
server_key_version: "encryptionkey@0"
I20260810 23:37:12.965258 9217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance:
uuid: "4a66cb6c53404d718ed33403179eb5c7"
format_stamp: "Formatted at 2026-08-10 23:37:12 on dist-test-slave-04bb"
server_key: "059afeda6f8d13ebc983965a11d5d56c"
server_key_iv: "a9ec3e71a627b14842ce71924ff94c20"
server_key_version: "encryptionkey@0"
I20260810 23:37:12.970530 9217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:37:12.974237 9243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:12.975915 9217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:37:12.976068 9217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/wal
uuid: "4a66cb6c53404d718ed33403179eb5c7"
format_stamp: "Formatted at 2026-08-10 23:37:12 on dist-test-slave-04bb"
server_key: "059afeda6f8d13ebc983965a11d5d56c"
server_key_iv: "a9ec3e71a627b14842ce71924ff94c20"
server_key_version: "encryptionkey@0"
I20260810 23:37:12.976307 9217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:12.992071 9217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:12.993121 9217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:12.993430 9217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:12.994592 9217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:37:12.996510 9217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:37:12.996724 9217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 23:37:12.996850 9217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:37:12.996940 9217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:13.041698 9217 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.194:37431
I20260810 23:37:13.041724 9427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.194:37431 every 8 connection(s)
I20260810 23:37:13.043100 9217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/data/info.pb
I20260810 23:37:13.049187 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 9217
I20260810 23:37:13.049388 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/wal/instance
I20260810 23:37:13.049963 9411 external_mini_cluster.cc:1428] Setting key 2fb0d4f045a739c1e3a9bc703bffff46
I20260810 23:37:13.052855 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.48.195:0
--local_ip_for_outbound_sockets=127.9.48.195
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33967
--builtin_ntp_servers=127.9.48.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:37:13.056816 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.045939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.194:50601 (local address 127.9.48.254:33967)
0810 23:37:13.046161 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:13.046168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:13.047495 (+ 1327us) server_negotiation.cc:407] Connection header received
0810 23:37:13.049503 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:13.049509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:13.049602 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:13.049731 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:13.050972 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.051863 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.053619 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.054362 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.054512 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.055828 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:13.055836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:13.055844 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:13.055870 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:13.055911 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:13.055929 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:13.056023 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:13.056267 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 23:37:13.056379 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:37:13.057686 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.046423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33967 (local address 127.9.48.194:50601)
0810 23:37:13.047348 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:13.047385 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:13.049149 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:13.049920 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:13.049928 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:13.050042 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:13.050814 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:13.050836 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.052045 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.052049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:13.053469 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:13.053482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.054576 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.054581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:13.054784 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.055497 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:37:13.055500 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:13.055538 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:37:13.055631 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:13.056028 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:13.056037 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:13.056042 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:37:13.056216 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 23:37:13.056487 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":304,"threads_started":1}
I20260810 23:37:13.059199 9432 heartbeater.cc:347] Connected to a master server at 127.9.48.254:33967
I20260810 23:37:13.059528 9432 heartbeater.cc:464] Registering TS with master...
I20260810 23:37:13.060292 9432 heartbeater.cc:511] Master 127.9.48.254:33967 requested a full tablet report, sending...
I20260810 23:37:13.061854 8921 ts_manager.cc:194] Registered new tserver with Master: 4a66cb6c53404d718ed33403179eb5c7 (127.9.48.194:37431)
I20260810 23:37:13.062747 8921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.48.194:50601
W20260810 23:37:13.398082 9435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:13.398361 9435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:13.398449 9435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:13.405524 9435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:13.405622 9435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:13.405733 9435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.48.195
I20260810 23:37:13.414647 9435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.48.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.48.195
--webserver_port=0
--tserver_master_addrs=127.9.48.254:33967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.48.195
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:13.416003 9435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:13.417932 9435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:13.430979 9440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:13.431026 9446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:13.431051 9441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:13.432008 9435 server_base.cc:1034] running on GCE node
I20260810 23:37:13.432682 9435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:13.433696 9435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:13.434967 9435 hybrid_clock.cc:648] HybridClock initialized: now 1786405033434901 us; error 49 us; skew 500 ppm
I20260810 23:37:13.435366 9435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 23:37:13.437983 9435 webserver.cc:460] Webserver started at http://127.9.48.195:45599/ using document root <none> and password file <none>
I20260810 23:37:13.438737 9435 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:13.438874 9435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:13.439221 9435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:13.442162 9435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/data/instance:
uuid: "e7285e3ef96746f6aa4150dac1753cdc"
format_stamp: "Formatted at 2026-08-10 23:37:13 on dist-test-slave-04bb"
server_key: "1249b24162cf1e7db0c5f78d84b2e242"
server_key_iv: "bdda2b89b98887e11ab6e0a7aedb5215"
server_key_version: "encryptionkey@0"
I20260810 23:37:13.442977 9435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance:
uuid: "e7285e3ef96746f6aa4150dac1753cdc"
format_stamp: "Formatted at 2026-08-10 23:37:13 on dist-test-slave-04bb"
server_key: "1249b24162cf1e7db0c5f78d84b2e242"
server_key_iv: "bdda2b89b98887e11ab6e0a7aedb5215"
server_key_version: "encryptionkey@0"
I20260810 23:37:13.447953 9435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:37:13.451418 9459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:13.452724 9435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:37:13.452888 9435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/wal
uuid: "e7285e3ef96746f6aa4150dac1753cdc"
format_stamp: "Formatted at 2026-08-10 23:37:13 on dist-test-slave-04bb"
server_key: "1249b24162cf1e7db0c5f78d84b2e242"
server_key_iv: "bdda2b89b98887e11ab6e0a7aedb5215"
server_key_version: "encryptionkey@0"
I20260810 23:37:13.453118 9435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:13.481717 9435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:13.482681 9435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:13.483001 9435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:13.484040 9435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:37:13.485648 9435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:37:13.485783 9435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:13.485913 9435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:37:13.486014 9435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:13.520428 9435 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.195:36071
I20260810 23:37:13.520537 9639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.195:36071 every 8 connection(s)
I20260810 23:37:13.521806 9435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/data/info.pb
I20260810 23:37:13.527554 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 9435
I20260810 23:37:13.527760 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/wal/instance
I20260810 23:37:13.528239 9411 external_mini_cluster.cc:1428] Setting key 3863986b48e534579aefdda7ae98c868
I20260810 23:37:13.535605 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.524586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.195:41199 (local address 127.9.48.254:33967)
0810 23:37:13.524795 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:13.524800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:13.526396 (+ 1596us) server_negotiation.cc:407] Connection header received
0810 23:37:13.528678 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:13.528684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:13.528795 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:13.528925 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:13.530220 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.531110 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.532936 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.533355 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.533454 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.534815 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:13.534820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:13.534824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:13.534841 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:13.534873 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:13.534885 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:13.534965 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:13.535237 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 23:37:13.535364 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 23:37:13.536711 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.525138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:33967 (local address 127.9.48.195:41199)
0810 23:37:13.526195 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:13.526245 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:13.528300 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:13.529279 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:13.529289 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:13.529394 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:13.530021 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:13.530039 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.531281 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.531287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:13.532787 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:13.532800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.533563 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.533568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:13.533797 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.534470 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:37:13.534473 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:13.534514 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:37:13.534605 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:13.534984 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:13.534993 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:13.534999 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:37:13.535183 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 23:37:13.535483 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":340,"threads_started":1}
I20260810 23:37:13.538290 9640 heartbeater.cc:347] Connected to a master server at 127.9.48.254:33967
I20260810 23:37:13.538568 9640 heartbeater.cc:464] Registering TS with master...
I20260810 23:37:13.539263 9640 heartbeater.cc:511] Master 127.9.48.254:33967 requested a full tablet report, sending...
I20260810 23:37:13.540875 8921 ts_manager.cc:194] Registered new tserver with Master: e7285e3ef96746f6aa4150dac1753cdc (127.9.48.195:36071)
I20260810 23:37:13.541742 8921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.48.195:41199
I20260810 23:37:13.543982 9411 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:37:13.555944 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.548594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.9.48.254:33967)
0810 23:37:13.548867 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:13.548877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:13.549233 (+ 356us) server_negotiation.cc:407] Connection header received
0810 23:37:13.549456 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:13.549469 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:13.549543 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:13.549670 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:13.550292 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.551076 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.553270 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.553753 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.553910 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.555120 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:13.555126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:13.555130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:13.555155 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:13.555188 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:13.555203 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:13.555315 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:13.555501 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 23:37:13.555602 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 23:37:13.560302 8921 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:37:13.567193 8921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:37:13.579787 9210 heartbeater.cc:503] Master 127.9.48.254:33967 was elected leader, sending a full tablet report...
I20260810 23:37:13.594560 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.580133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54702 (local address 127.9.48.193:42625)
0810 23:37:13.582676 (+ 2543us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:13.582690 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:13.585707 (+ 3017us) server_negotiation.cc:407] Connection header received
0810 23:37:13.586022 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:13.586032 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:13.586269 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:13.586440 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:13.587452 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.589064 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.591786 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.593126 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.593275 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.593498 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:13.593980 (+ 482us) server_negotiation.cc:299] Negotiation successful
0810 23:37:13.594145 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2335,"thread_start_us":2138,"threads_started":1}
I20260810 23:37:13.594895 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.579401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:36071 (local address 127.0.0.1:48230)
0810 23:37:13.583438 (+ 4037us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:13.583492 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:13.583701 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:13.586115 (+ 2414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:13.586140 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:13.586185 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:13.586655 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:13.586668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.588578 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.588584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:13.591291 (+ 2707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:13.591309 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.592968 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.592975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:13.593193 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.593250 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 23:37:13.593381 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 23:37:13.594508 (+ 1127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3695,"mutex_wait_us":127,"thread_start_us":6985,"threads_started":1}
I20260810 23:37:13.594228 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.579599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48230 (local address 127.9.48.195:36071)
0810 23:37:13.584938 (+ 5339us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:13.584950 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:13.584999 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:37:13.585152 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:13.585165 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:13.585481 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:13.585712 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:13.586920 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.588406 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.591512 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.592774 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.592939 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.593233 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:13.593660 (+ 427us) server_negotiation.cc:299] Negotiation successful
0810 23:37:13.593834 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5098,"thread_start_us":4894,"threads_started":1}
I20260810 23:37:13.595094 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.582597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.9.48.194:37431)
0810 23:37:13.585124 (+ 2527us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:13.585137 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:13.586996 (+ 1859us) server_negotiation.cc:407] Connection header received
0810 23:37:13.587099 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:13.587108 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:13.587343 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:13.587524 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:13.588354 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.590053 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.592246 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.593678 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.593823 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.594037 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:13.594548 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 23:37:13.594712 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2341,"thread_start_us":2230,"threads_started":1}
I20260810 23:37:13.595206 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.582029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:37431 (local address 127.0.0.1:41730)
0810 23:37:13.586734 (+ 4705us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:13.586750 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:13.586859 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:13.587769 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:13.587773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:13.587798 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:13.588146 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:13.588158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.590214 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.590220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:13.592038 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:13.592053 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.594181 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.594188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:13.594358 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.594390 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:37:13.594505 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 23:37:13.594696 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4536,"mutex_wait_us":28,"thread_start_us":3267,"threads_started":1}
I20260810 23:37:13.596079 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.579463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:42625 (local address 127.0.0.1:54702)
0810 23:37:13.585594 (+ 6131us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:13.585611 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:13.585824 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:13.586758 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:13.586764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:13.586796 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:13.587255 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:13.587269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.589243 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.589252 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:13.591605 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:13.591616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.593554 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.593563 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:13.593759 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.593796 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:37:13.595613 (+ 1817us) client_negotiation.cc:240] Negotiation successful
0810 23:37:13.595764 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5982,"mutex_wait_us":24,"thread_start_us":3437,"threads_started":1}
I20260810 23:37:13.599503 9318 tablet_service.cc:1467] Processing CreateTablet for tablet 761f4284854346f79e270b09522d8638 (DEFAULT_TABLE table=test-table [id=264870b12fde410ea65cb159420558e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:37:13.599499 9537 tablet_service.cc:1467] Processing CreateTablet for tablet 761f4284854346f79e270b09522d8638 (DEFAULT_TABLE table=test-table [id=264870b12fde410ea65cb159420558e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:37:13.599802 9111 tablet_service.cc:1467] Processing CreateTablet for tablet 761f4284854346f79e270b09522d8638 (DEFAULT_TABLE table=test-table [id=264870b12fde410ea65cb159420558e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:37:13.601027 9537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761f4284854346f79e270b09522d8638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:13.601027 9318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761f4284854346f79e270b09522d8638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:13.601075 9111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761f4284854346f79e270b09522d8638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:13.613401 9667 tablet_bootstrap.cc:492] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2: Bootstrap starting.
I20260810 23:37:13.615793 9669 tablet_bootstrap.cc:492] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7: Bootstrap starting.
I20260810 23:37:13.617347 9670 tablet_bootstrap.cc:492] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc: Bootstrap starting.
I20260810 23:37:13.618002 9667 tablet_bootstrap.cc:654] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:13.619375 9667 log.cc:824] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:13.619557 9669 tablet_bootstrap.cc:654] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:13.620857 9670 tablet_bootstrap.cc:654] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:13.620939 9669 log.cc:824] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:13.621969 9670 log.cc:824] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:13.623579 9667 tablet_bootstrap.cc:492] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2: No bootstrap required, opened a new log
I20260810 23:37:13.623941 9667 ts_tablet_manager.cc:1397] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260810 23:37:13.624581 9670 tablet_bootstrap.cc:492] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc: No bootstrap required, opened a new log
I20260810 23:37:13.624874 9670 ts_tablet_manager.cc:1397] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260810 23:37:13.630256 9669 tablet_bootstrap.cc:492] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7: No bootstrap required, opened a new log
I20260810 23:37:13.630533 9669 ts_tablet_manager.cc:1397] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260810 23:37:13.632951 9670 raft_consensus.cc:348] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7285e3ef96746f6aa4150dac1753cdc" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 36071 } } peers { permanent_uuid: "7e8e4b1eb75d4d36925424942571c4c2" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 42625 } } peers { permanent_uuid: "4a66cb6c53404d718ed33403179eb5c7" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37431 } }
I20260810 23:37:13.633340 9670 raft_consensus.cc:374] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:13.633471 9670 raft_consensus.cc:729] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7285e3ef96746f6aa4150dac1753cdc, State: Initialized, Role: FOLLOWER
I20260810 23:37:13.634094 9670 consensus_queue.cc:260] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7285e3ef96746f6aa4150dac1753cdc" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 36071 } } peers { permanent_uuid: "7e8e4b1eb75d4d36925424942571c4c2" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 42625 } } peers { permanent_uuid: "4a66cb6c53404d718ed33403179eb5c7" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37431 } }
I20260810 23:37:13.635227 9640 heartbeater.cc:503] Master 127.9.48.254:33967 was elected leader, sending a full tablet report...
I20260810 23:37:13.635973 9670 ts_tablet_manager.cc:1428] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260810 23:37:13.635761 9667 raft_consensus.cc:348] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7285e3ef96746f6aa4150dac1753cdc" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 36071 } } peers { permanent_uuid: "7e8e4b1eb75d4d36925424942571c4c2" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 42625 } } peers { permanent_uuid: "4a66cb6c53404d718ed33403179eb5c7" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37431 } }
I20260810 23:37:13.636188 9667 raft_consensus.cc:374] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:13.636286 9667 raft_consensus.cc:729] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e8e4b1eb75d4d36925424942571c4c2, State: Initialized, Role: FOLLOWER
I20260810 23:37:13.636925 9667 consensus_queue.cc:260] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7285e3ef96746f6aa4150dac1753cdc" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 36071 } } peers { permanent_uuid: "7e8e4b1eb75d4d36925424942571c4c2" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 42625 } } peers { permanent_uuid: "4a66cb6c53404d718ed33403179eb5c7" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37431 } }
I20260810 23:37:13.638680 9667 ts_tablet_manager.cc:1428] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260810 23:37:13.641198 9669 raft_consensus.cc:348] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7285e3ef96746f6aa4150dac1753cdc" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 36071 } } peers { permanent_uuid: "7e8e4b1eb75d4d36925424942571c4c2" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 42625 } } peers { permanent_uuid: "4a66cb6c53404d718ed33403179eb5c7" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37431 } }
I20260810 23:37:13.641574 9669 raft_consensus.cc:374] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:13.641695 9669 raft_consensus.cc:729] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a66cb6c53404d718ed33403179eb5c7, State: Initialized, Role: FOLLOWER
I20260810 23:37:13.642288 9669 consensus_queue.cc:260] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7285e3ef96746f6aa4150dac1753cdc" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 36071 } } peers { permanent_uuid: "7e8e4b1eb75d4d36925424942571c4c2" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 42625 } } peers { permanent_uuid: "4a66cb6c53404d718ed33403179eb5c7" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37431 } }
I20260810 23:37:13.643383 9432 heartbeater.cc:503] Master 127.9.48.254:33967 was elected leader, sending a full tablet report...
I20260810 23:37:13.643956 9669 ts_tablet_manager.cc:1428] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260810 23:37:13.757694 9678 raft_consensus.cc:482] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:37:13.758065 9678 raft_consensus.cc:504] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7285e3ef96746f6aa4150dac1753cdc" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 36071 } } peers { permanent_uuid: "7e8e4b1eb75d4d36925424942571c4c2" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 42625 } } peers { permanent_uuid: "4a66cb6c53404d718ed33403179eb5c7" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37431 } }
I20260810 23:37:13.759931 9678 leader_election.cc:288] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7285e3ef96746f6aa4150dac1753cdc (127.9.48.195:36071), 4a66cb6c53404d718ed33403179eb5c7 (127.9.48.194:37431)
I20260810 23:37:13.766882 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.760376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:52517 (local address 127.9.48.194:37431)
0810 23:37:13.760666 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:13.760675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:13.761357 (+ 682us) server_negotiation.cc:407] Connection header received
0810 23:37:13.761595 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:13.761610 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:13.761714 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:13.761814 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:13.762463 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.763200 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.765223 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.765970 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.766146 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.766218 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:13.766486 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 23:37:13.766627 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:37:13.766978 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.760269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.48.193:36201 (local address 127.9.48.195:36071)
0810 23:37:13.760472 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:13.760480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:13.761583 (+ 1103us) server_negotiation.cc:407] Connection header received
0810 23:37:13.761779 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:13.761783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:13.761843 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:13.761942 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:37:13.762605 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.763316 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.765295 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.766093 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.766270 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.766337 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:13.766607 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 23:37:13.766742 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:37:13.767196 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.760357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.194:37431 (local address 127.9.48.193:52517)
0810 23:37:13.761216 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:13.761240 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:13.761432 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:13.761842 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:13.761846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:13.761870 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:13.762306 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:13.762320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.763356 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.763360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:13.765021 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:13.765036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.766168 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.766174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:13.766326 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.766350 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:37:13.766457 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 23:37:13.766586 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":26,"thread_start_us":392,"threads_started":1}
I20260810 23:37:13.767279 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.760364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.195:36071 (local address 127.9.48.193:36201)
0810 23:37:13.761474 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:13.761488 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:13.761637 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:13.761966 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:13.761970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:13.762011 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:37:13.762425 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:13.762434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.763451 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.763457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:13.765078 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:13.765091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.766295 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.766301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:13.766451 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.766474 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:37:13.766578 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 23:37:13.766698 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":328,"thread_start_us":487,"threads_started":1}
I20260810 23:37:13.768237 9351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761f4284854346f79e270b09522d8638" candidate_uuid: "7e8e4b1eb75d4d36925424942571c4c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a66cb6c53404d718ed33403179eb5c7" is_pre_election: true
I20260810 23:37:13.768240 9568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761f4284854346f79e270b09522d8638" candidate_uuid: "7e8e4b1eb75d4d36925424942571c4c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7285e3ef96746f6aa4150dac1753cdc" is_pre_election: true
I20260810 23:37:13.768777 9351 raft_consensus.cc:2454] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e8e4b1eb75d4d36925424942571c4c2 in term 0.
I20260810 23:37:13.768783 9568 raft_consensus.cc:2454] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e8e4b1eb75d4d36925424942571c4c2 in term 0.
I20260810 23:37:13.769441 9041 leader_election.cc:302] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e8e4b1eb75d4d36925424942571c4c2, e7285e3ef96746f6aa4150dac1753cdc; no voters:
I20260810 23:37:13.769860 9678 raft_consensus.cc:2781] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:37:13.770004 9678 raft_consensus.cc:482] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:37:13.770120 9678 raft_consensus.cc:3037] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:13.772480 9678 raft_consensus.cc:504] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7285e3ef96746f6aa4150dac1753cdc" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 36071 } } peers { permanent_uuid: "7e8e4b1eb75d4d36925424942571c4c2" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 42625 } } peers { permanent_uuid: "4a66cb6c53404d718ed33403179eb5c7" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37431 } }
I20260810 23:37:13.773437 9678 leader_election.cc:288] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [CANDIDATE]: Term 1 election: Requested vote from peers e7285e3ef96746f6aa4150dac1753cdc (127.9.48.195:36071), 4a66cb6c53404d718ed33403179eb5c7 (127.9.48.194:37431)
W20260810 23:37:13.773665 9642 tablet.cc:2369] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:37:13.773705 9568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761f4284854346f79e270b09522d8638" candidate_uuid: "7e8e4b1eb75d4d36925424942571c4c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7285e3ef96746f6aa4150dac1753cdc"
I20260810 23:37:13.773968 9568 raft_consensus.cc:3037] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:13.774842 9351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761f4284854346f79e270b09522d8638" candidate_uuid: "7e8e4b1eb75d4d36925424942571c4c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a66cb6c53404d718ed33403179eb5c7"
I20260810 23:37:13.775038 9351 raft_consensus.cc:3037] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:13.776338 9568 raft_consensus.cc:2454] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e8e4b1eb75d4d36925424942571c4c2 in term 1.
I20260810 23:37:13.776757 9041 leader_election.cc:302] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e8e4b1eb75d4d36925424942571c4c2, e7285e3ef96746f6aa4150dac1753cdc; no voters:
I20260810 23:37:13.777063 9678 raft_consensus.cc:2781] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:13.777505 9351 raft_consensus.cc:2454] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e8e4b1eb75d4d36925424942571c4c2 in term 1.
I20260810 23:37:13.777604 9678 raft_consensus.cc:686] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [term 1 LEADER]: Becoming Leader. State: Replica: 7e8e4b1eb75d4d36925424942571c4c2, State: Running, Role: LEADER
I20260810 23:37:13.778409 9678 consensus_queue.cc:237] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7285e3ef96746f6aa4150dac1753cdc" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 36071 } } peers { permanent_uuid: "7e8e4b1eb75d4d36925424942571c4c2" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 42625 } } peers { permanent_uuid: "4a66cb6c53404d718ed33403179eb5c7" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37431 } }
I20260810 23:37:13.783213 8921 catalog_manager.cc:5495] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e8e4b1eb75d4d36925424942571c4c2 (127.9.48.193). New cstate: current_term: 1 leader_uuid: "7e8e4b1eb75d4d36925424942571c4c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7285e3ef96746f6aa4150dac1753cdc" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 36071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e8e4b1eb75d4d36925424942571c4c2" member_type: VOTER last_known_addr { host: "127.9.48.193" port: 42625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a66cb6c53404d718ed33403179eb5c7" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37431 } health_report { overall_health: UNKNOWN } } }
W20260810 23:37:13.795342 9433 tablet.cc:2369] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:37:13.805822 9212 tablet.cc:2369] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:37:13.832439 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.827058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.9.48.193:42625)
0810 23:37:13.827299 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:13.827304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:13.827323 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:37:13.827466 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:13.827470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:13.827563 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:13.827712 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:37:13.828052 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.828876 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.830365 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.830789 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.830882 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.831360 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:37:13.831908 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:37:13.832045 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:13.832126 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 23:37:13.832222 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 23:37:13.841169 9351 raft_consensus.cc:1261] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7 [term 1 FOLLOWER]: Refusing update from remote peer 7e8e4b1eb75d4d36925424942571c4c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:37:13.841161 9568 raft_consensus.cc:1261] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc [term 1 FOLLOWER]: Refusing update from remote peer 7e8e4b1eb75d4d36925424942571c4c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:37:13.842226 9684 consensus_queue.cc:1035] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a66cb6c53404d718ed33403179eb5c7" member_type: VOTER last_known_addr { host: "127.9.48.194" port: 37431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:37:13.842594 9678 consensus_queue.cc:1035] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7285e3ef96746f6aa4150dac1753cdc" member_type: VOTER last_known_addr { host: "127.9.48.195" port: 36071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:37:13.870097 8920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:37:13.870481 8920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:37:13.875161 8920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c18fd87a0d4a486884ef47f6e6d3ab51: Sending DeleteTablet for 3 replicas of tablet 761f4284854346f79e270b09522d8638
I20260810 23:37:13.876622 9537 tablet_service.cc:1514] Processing DeleteTablet for tablet 761f4284854346f79e270b09522d8638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:37:13 UTC) from {username='slave'} at 127.0.0.1:48230
I20260810 23:37:13.877063 9318 tablet_service.cc:1514] Processing DeleteTablet for tablet 761f4284854346f79e270b09522d8638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:37:13 UTC) from {username='slave'} at 127.0.0.1:41730
I20260810 23:37:13.877450 9111 tablet_service.cc:1514] Processing DeleteTablet for tablet 761f4284854346f79e270b09522d8638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:37:13 UTC) from {username='slave'} at 127.0.0.1:54702
I20260810 23:37:13.877712 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 9020
I20260810 23:37:13.889314 9713 tablet_replica.cc:330] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2: stopping tablet replica
I20260810 23:37:13.890025 9713 raft_consensus.cc:2229] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:37:13.891425 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:13.879011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.9.48.193:42625)
0810 23:37:13.879223 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:13.879230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:13.879397 (+ 167us) server_negotiation.cc:407] Connection header received
0810 23:37:13.879646 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:13.879652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:13.879758 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:13.879871 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:13.880362 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.881681 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.886489 (+ 4808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:13.887227 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:13.887348 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:13.889087 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:13.889102 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:13.889134 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:13.890072 (+ 938us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:13.890142 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:13.890178 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:13.890275 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:13.890718 (+ 443us) server_negotiation.cc:299] Negotiation successful
0810 23:37:13.890845 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:37:13.891880 9712 tablet_replica.cc:330] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7: stopping tablet replica
I20260810 23:37:13.892418 9712 raft_consensus.cc:2229] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:37:13.892721 9202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:13.893081 9712 raft_consensus.cc:2258] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:37:13.895154 9712 ts_tablet_manager.cc:1905] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:37:13.900193 9712 ts_tablet_manager.cc:1918] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:37:13.900383 9712 log.cc:1196] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-1/wal/wals/761f4284854346f79e270b09522d8638
I20260810 23:37:13.900453 9710 tablet_replica.cc:330] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc: stopping tablet replica
I20260810 23:37:13.901077 9712 ts_tablet_manager.cc:1939] T 761f4284854346f79e270b09522d8638 P 4a66cb6c53404d718ed33403179eb5c7: Deleting consensus metadata
I20260810 23:37:13.901031 9710 raft_consensus.cc:2229] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:37:13.901857 9710 raft_consensus.cc:2258] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:37:13.903367 8900 catalog_manager.cc:4841] TS 4a66cb6c53404d718ed33403179eb5c7 (127.9.48.194:37431): tablet 761f4284854346f79e270b09522d8638 (table test-table [id=264870b12fde410ea65cb159420558e2]) successfully deleted
I20260810 23:37:13.904181 9710 ts_tablet_manager.cc:1905] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:37:13.909045 9710 ts_tablet_manager.cc:1918] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:37:13.909226 9710 log.cc:1196] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc: Deleting WAL directory at /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786404698557218-9411-0/minicluster-data/ts-2/wal/wals/761f4284854346f79e270b09522d8638
I20260810 23:37:13.909721 9710 ts_tablet_manager.cc:1939] T 761f4284854346f79e270b09522d8638 P e7285e3ef96746f6aa4150dac1753cdc: Deleting consensus metadata
I20260810 23:37:13.911320 8897 catalog_manager.cc:4841] TS e7285e3ef96746f6aa4150dac1753cdc (127.9.48.195:36071): tablet 761f4284854346f79e270b09522d8638 (table test-table [id=264870b12fde410ea65cb159420558e2]) successfully deleted
I20260810 23:37:13.890947 9713 raft_consensus.cc:2258] T 761f4284854346f79e270b09522d8638 P 7e8e4b1eb75d4d36925424942571c4c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:37:13.997956 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 9020
W20260810 23:37:14.024616 8903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.48.193:42625 (error 108)
I20260810 23:37:14.024817 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 9217
I20260810 23:37:14.026772 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:14.026348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:42625 (local address 127.0.0.1:54730)
0810 23:37:14.026517 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:14.026631 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 23:37:14.027029 8903 catalog_manager.cc:4584] TS 7e8e4b1eb75d4d36925424942571c4c2 (127.9.48.193:42625): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 761f4284854346f79e270b09522d8638: Network error: Client connection negotiation failed: client connection to 127.9.48.193:42625: connect: Connection refused (error 111)
I20260810 23:37:14.032196 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:14.025767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.9.48.194:37431)
0810 23:37:14.025959 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:14.025966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:14.025991 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:37:14.026188 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:14.026194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:14.026296 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:14.026430 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:14.026849 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:14.027723 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:14.029076 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:14.029565 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:14.029701 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:14.030825 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:14.030835 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:14.030848 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:14.031420 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:14.031465 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:14.031485 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:14.031622 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:14.031758 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 23:37:14.031865 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:37:14.033046 9422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:14.080884 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:14.080338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:42625 (local address 127.0.0.1:54742)
0810 23:37:14.080606 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:14.080696 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 23:37:14.081961 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:14.081594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:42625 (local address 127.0.0.1:54748)
0810 23:37:14.081772 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:14.081853 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 23:37:14.124635 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 9217
I20260810 23:37:14.142534 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:14.142059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:42625 (local address 127.0.0.1:54760)
0810 23:37:14.142268 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:14.142370 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 23:37:14.143594 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:14.143247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:42625 (local address 127.0.0.1:54766)
0810 23:37:14.143399 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:14.143477 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:37:14.153615 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 9435
I20260810 23:37:14.160411 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:14.154403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.9.48.195:36071)
0810 23:37:14.154572 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:14.154577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:14.154596 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:37:14.154750 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:14.154755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:14.154844 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:14.154942 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:14.155304 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:14.156041 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:14.157371 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:14.157872 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:14.157969 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:14.159020 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:14.159029 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:14.159044 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:14.159727 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:14.159775 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:14.159796 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:14.159893 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:14.160030 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 23:37:14.160115 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:37:14.161124 9633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:14.224690 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:14.224085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:42625 (local address 127.0.0.1:54770)
0810 23:37:14.224344 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:14.224437 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 23:37:14.225821 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:14.225415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.193:42625 (local address 127.0.0.1:54786)
0810 23:37:14.225639 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:14.225709 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.193:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 23:37:14.250304 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 9435
I20260810 23:37:14.276437 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 8878
I20260810 23:37:14.277662 8969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:14.388245 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 8878
2026-08-10T23:37:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2963 ms)
[----------] 6 tests from EncryptionPolicyTest (23642 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:14 dist-test-slave-04bb krb5kdc[9725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:14 dist-test-slave-04bb krb5kdc[9725](info): set up 2 sockets
Aug 10 23:37:14 dist-test-slave-04bb krb5kdc[9725](info): commencing operation
krb5kdc: starting...
W20260810 23:37:15.784721 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.341s user 0.004s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:16 dist-test-slave-04bb krb5kdc[9725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:37:16.128842 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:16.482385 9751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:16.482690 9751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:16.482892 9751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:16.490226 9751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:16.490350 9751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:16.490468 9751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:16.490586 9751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:16.490664 9751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:16.500613 9751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40537
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:16.501973 9751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:16.503412 9751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:16.516178 9765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:16.516350 9760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:16.516331 9763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:16.517410 9751 server_base.cc:1034] running on GCE node
I20260810 23:37:16.518270 9751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:16.519734 9751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:16.520956 9751 hybrid_clock.cc:648] HybridClock initialized: now 1786405036520912 us; error 35 us; skew 500 ppm
I20260810 23:37:16.521394 9751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:16 dist-test-slave-04bb krb5kdc[9725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:16.524735 9751 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:37:16.526432 9751 webserver.cc:460] Webserver started at http://127.0.0.1:36911/ using document root <none> and password file <none>
I20260810 23:37:16.527176 9751 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:16.527340 9751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:16.527681 9751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:16.530884 9751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "10ba9d8a1b5746b8bf4816ab1b08364a"
format_stamp: "Formatted at 2026-08-10 23:37:16 on dist-test-slave-04bb"
server_key: "1d259a1767b3f3dcf9bf4f7a601128e7"
server_key_iv: "358633588cd829893ef48c49a25d805c"
server_key_version: "encryptionkey@0"
I20260810 23:37:16.531726 9751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "10ba9d8a1b5746b8bf4816ab1b08364a"
format_stamp: "Formatted at 2026-08-10 23:37:16 on dist-test-slave-04bb"
server_key: "1d259a1767b3f3dcf9bf4f7a601128e7"
server_key_iv: "358633588cd829893ef48c49a25d805c"
server_key_version: "encryptionkey@0"
I20260810 23:37:16.537223 9751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:37:16.540977 9776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:16.542383 9751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:37:16.542598 9751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "10ba9d8a1b5746b8bf4816ab1b08364a"
format_stamp: "Formatted at 2026-08-10 23:37:16 on dist-test-slave-04bb"
server_key: "1d259a1767b3f3dcf9bf4f7a601128e7"
server_key_iv: "358633588cd829893ef48c49a25d805c"
server_key_version: "encryptionkey@0"
I20260810 23:37:16.542840 9751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:16.580776 9751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:16.585040 9751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:16.585355 9751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:16.604753 9751 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40537
I20260810 23:37:16.604803 9869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40537 every 8 connection(s)
I20260810 23:37:16.606179 9751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:16.609985 9871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:16.611385 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 9751
I20260810 23:37:16.611620 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:16.612483 9411 external_mini_cluster.cc:1428] Setting key 370fb03d4d99d9f6d39565504a3b02cd
I20260810 23:37:16.624915 9871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a: Bootstrap starting.
Aug 10 23:37:16 dist-test-slave-04bb krb5kdc[9725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:37:16.628759 9871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:16.630177 9871 log.cc:824] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:16.633769 9871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a: No bootstrap required, opened a new log
I20260810 23:37:16.638074 9874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:16.614541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56561 (local address 127.0.0.1:40537)
0810 23:37:16.615391 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:16.615402 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:16.615449 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:37:16.617099 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:16.617121 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:16.617526 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:16.617847 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:16.619748 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:16.621172 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:16.623153 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:16.623816 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:16.623943 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:16.629059 (+ 5116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:16.629072 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:16.629100 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:16.629155 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:16.633312 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:16.634198 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:16.634209 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:16.634224 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:16.634443 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:16.634901 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:16.634906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:16.634909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:16.635603 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:16.635946 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:16.636337 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 23:37:16.636698 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":412,"threads_started":1}
I20260810 23:37:16.642282 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:16.643641 9871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ba9d8a1b5746b8bf4816ab1b08364a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } }
I20260810 23:37:16.643992 9871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:16.644124 9871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ba9d8a1b5746b8bf4816ab1b08364a, State: Initialized, Role: FOLLOWER
I20260810 23:37:16.644767 9871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ba9d8a1b5746b8bf4816ab1b08364a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } }
I20260810 23:37:16.645002 9871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:16.645138 9871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:16.645319 9871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:16.647631 9871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ba9d8a1b5746b8bf4816ab1b08364a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } }
I20260810 23:37:16.648118 9871 leader_election.cc:302] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10ba9d8a1b5746b8bf4816ab1b08364a; no voters:
I20260810 23:37:16.648814 9871 leader_election.cc:288] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:16.648929 9878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:16.651371 9871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:16.655547 9878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [term 1 LEADER]: Becoming Leader. State: Replica: 10ba9d8a1b5746b8bf4816ab1b08364a, State: Running, Role: LEADER
I20260810 23:37:16.656248 9878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ba9d8a1b5746b8bf4816ab1b08364a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } }
I20260810 23:37:16.659963 9879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10ba9d8a1b5746b8bf4816ab1b08364a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ba9d8a1b5746b8bf4816ab1b08364a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } } }
I20260810 23:37:16.660430 9879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:16.659991 9881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10ba9d8a1b5746b8bf4816ab1b08364a. Latest consensus state: current_term: 1 leader_uuid: "10ba9d8a1b5746b8bf4816ab1b08364a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ba9d8a1b5746b8bf4816ab1b08364a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } } }
I20260810 23:37:16.661164 9881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:16.668383 9896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:16.673795 9896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:16.685037 9896 catalog_manager.cc:1347] Generated new cluster ID: 99392c1990a448cca83471b15146efd4
I20260810 23:37:16.685149 9896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:16.687999 9874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:16.662407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33987 (local address 127.0.0.1:40537)
0810 23:37:16.662650 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:16.662660 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:16.667573 (+ 4913us) server_negotiation.cc:407] Connection header received
0810 23:37:16.668818 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:16.668827 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:16.668939 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:16.669077 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:16.671276 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:16.672102 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:16.674042 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:16.675025 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:16.675194 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:16.679597 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:16.679608 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:16.679616 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:16.679669 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:16.684059 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:16.685172 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:16.685177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:16.685181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:16.685331 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:16.685943 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:16.685949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:16.685952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:16.686443 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:16.686728 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:16.687151 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 23:37:16.687581 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:37:16.710986 9896 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:16.712879 9896 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:16.730855 9896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10ba9d8a1b5746b8bf4816ab1b08364a: Generated new TSK 0
I20260810 23:37:16.731956 9896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:16.739221 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 9751
I20260810 23:37:16.740584 9863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:16.836357 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 9751
W20260810 23:37:16.863963 9723 connection.cc:535] client connection to 127.0.0.1:40537 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40537): Connection reset by peer (error 104)
2026-08-10T23:37:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:16 dist-test-slave-04bb krb5kdc[9930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:16 dist-test-slave-04bb krb5kdc[9930](info): set up 2 sockets
Aug 10 23:37:16 dist-test-slave-04bb krb5kdc[9930](info): commencing operation
krb5kdc: starting...
W20260810 23:37:18.209050 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.325s user 0.001s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:18 dist-test-slave-04bb krb5kdc[9930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:37:18.536877 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45433
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:18.882370 9954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:18.882684 9954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:18.882890 9954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:18.890151 9954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:18.890262 9954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:18.890354 9954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:18.890446 9954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:18.890533 9954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:18.901922 9954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45433
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:18.903302 9954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:18.904765 9954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:18.917059 9961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:18.917104 9962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:18.917150 9965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:18.917981 9954 server_base.cc:1034] running on GCE node
I20260810 23:37:18.918848 9954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:18.920276 9954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:18.921511 9954 hybrid_clock.cc:648] HybridClock initialized: now 1786405038921453 us; error 69 us; skew 500 ppm
I20260810 23:37:18.921883 9954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:18 dist-test-slave-04bb krb5kdc[9930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:18.925091 9954 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:37:18.926705 9954 webserver.cc:460] Webserver started at http://127.0.0.1:33215/ using document root <none> and password file <none>
I20260810 23:37:18.927446 9954 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:18.927608 9954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:18.927965 9954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:18.931216 9954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "58471be60ee64669a9c06fead69667b0"
format_stamp: "Formatted at 2026-08-10 23:37:18 on dist-test-slave-04bb"
server_key: "8fbaba635b6aaa5b21e7b3fc3ff73386"
server_key_iv: "8ffe077e23248791c466fc698b7e479b"
server_key_version: "encryptionkey@0"
I20260810 23:37:18.932027 9954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "58471be60ee64669a9c06fead69667b0"
format_stamp: "Formatted at 2026-08-10 23:37:18 on dist-test-slave-04bb"
server_key: "8fbaba635b6aaa5b21e7b3fc3ff73386"
server_key_iv: "8ffe077e23248791c466fc698b7e479b"
server_key_version: "encryptionkey@0"
I20260810 23:37:18.937525 9954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:37:18.941334 9976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:18.942754 9954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:37:18.942960 9954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "58471be60ee64669a9c06fead69667b0"
format_stamp: "Formatted at 2026-08-10 23:37:18 on dist-test-slave-04bb"
server_key: "8fbaba635b6aaa5b21e7b3fc3ff73386"
server_key_iv: "8ffe077e23248791c466fc698b7e479b"
server_key_version: "encryptionkey@0"
I20260810 23:37:18.943171 9954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:18.968298 9954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:18.972764 9954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:18.973059 9954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:18.992129 9954 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45433
I20260810 23:37:18.992167 10061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45433 every 8 connection(s)
I20260810 23:37:18.993553 9954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:18.997368 10062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:19.000000 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 9954
I20260810 23:37:19.000202 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:19.000785 9411 external_mini_cluster.cc:1428] Setting key a5909049714080710bcd99d615dd19ac
I20260810 23:37:19.009795 10062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0: Bootstrap starting.
I20260810 23:37:19.013573 10062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:19.014681 10062 log.cc:824] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:19.014994 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:19.002823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43009 (local address 127.0.0.1:45433)
0810 23:37:19.003705 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:19.003717 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:19.003759 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:37:19.005519 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:19.005543 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:19.005930 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:19.006307 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:19.006916 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:19.008285 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:19.010236 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:19.010881 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:19.011014 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:19.012392 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:19.012400 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:19.012412 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:19.013018 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:19.013087 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:19.013112 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:19.013286 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:19.013708 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 23:37:19.013914 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":442,"threads_started":1}
I20260810 23:37:19.018040 10062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0: No bootstrap required, opened a new log
I20260810 23:37:19.019284 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:19.028723 10062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58471be60ee64669a9c06fead69667b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45433 } }
I20260810 23:37:19.029233 10062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:19.029384 10062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58471be60ee64669a9c06fead69667b0, State: Initialized, Role: FOLLOWER
I20260810 23:37:19.030090 10062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58471be60ee64669a9c06fead69667b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45433 } }
I20260810 23:37:19.030407 10062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:19.030555 10062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:19.030750 10062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:19.031980 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:19.024812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37687 (local address 127.0.0.1:45433)
0810 23:37:19.024961 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:19.024965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:19.025357 (+ 392us) server_negotiation.cc:407] Connection header received
0810 23:37:19.025533 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:19.025537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:19.025597 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:19.025694 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:19.026254 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:19.027028 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:19.029035 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:19.029631 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:19.029789 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:19.031159 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:19.031165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:19.031170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:19.031196 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:19.031255 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:19.031271 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:19.031418 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:19.031665 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 23:37:19.031717 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 23:37:19.033835 10062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58471be60ee64669a9c06fead69667b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45433 } }
I20260810 23:37:19.034364 10062 leader_election.cc:302] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58471be60ee64669a9c06fead69667b0; no voters:
I20260810 23:37:19.034965 10062 leader_election.cc:288] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:19.035120 10080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:19.035959 10080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [term 1 LEADER]: Becoming Leader. State: Replica: 58471be60ee64669a9c06fead69667b0, State: Running, Role: LEADER
I20260810 23:37:19.036494 10080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58471be60ee64669a9c06fead69667b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45433 } }
I20260810 23:37:19.037233 10062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:19.039719 10081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58471be60ee64669a9c06fead69667b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58471be60ee64669a9c06fead69667b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45433 } } }
I20260810 23:37:19.040118 10081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:19.040874 10082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58471be60ee64669a9c06fead69667b0. Latest consensus state: current_term: 1 leader_uuid: "58471be60ee64669a9c06fead69667b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58471be60ee64669a9c06fead69667b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45433 } } }
I20260810 23:37:19.041217 10082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:19.042411 10088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:19.048908 10088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:19.059103 10088 catalog_manager.cc:1347] Generated new cluster ID: 81c14151cff7491b9416a13a65732eeb
I20260810 23:37:19.059214 10088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:19.082695 10088 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:19.084512 10088 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:19.093119 10088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58471be60ee64669a9c06fead69667b0: Generated new TSK 0
I20260810 23:37:19.094076 10088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:19.105244 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 9954
W20260810 23:37:19.106616 10053 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43009
W20260810 23:37:19.107472 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 9954: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:37:19.107681 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 9954
W20260810 23:37:19.134377 9924 connection.cc:535] client connection to 127.0.0.1:45433 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45433): Connection reset by peer (error 104)
2026-08-10T23:37:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2270 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:19 dist-test-slave-04bb krb5kdc[10114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:19 dist-test-slave-04bb krb5kdc[10114](info): set up 2 sockets
Aug 10 23:37:19 dist-test-slave-04bb krb5kdc[10114](info): commencing operation
krb5kdc: starting...
W20260810 23:37:20.546103 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.393s user 0.001s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:20 dist-test-slave-04bb krb5kdc[10114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:37:20.877424 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:21.230247 10130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:21.230545 10130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:21.230639 10130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:21.237774 10130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:21.237893 10130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:21.237942 10130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:21.238008 10130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:21.238055 10130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:21.247861 10130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34949
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:21.249217 10130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:21.250527 10130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:21.263041 10137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:21.263042 10138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:21.263139 10141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:21.264036 10130 server_base.cc:1034] running on GCE node
I20260810 23:37:21.264875 10130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:21.266284 10130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:21.267534 10130 hybrid_clock.cc:648] HybridClock initialized: now 1786405041267496 us; error 58 us; skew 500 ppm
I20260810 23:37:21.267928 10130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:21 dist-test-slave-04bb krb5kdc[10114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:21.271492 10130 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:37:21.273109 10130 webserver.cc:460] Webserver started at http://127.0.0.1:39109/ using document root <none> and password file <none>
I20260810 23:37:21.274170 10130 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:21.274343 10130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:21.274757 10130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:21.277997 10130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "567923278e4b471fabaa9b0aa14b05af"
format_stamp: "Formatted at 2026-08-10 23:37:21 on dist-test-slave-04bb"
server_key: "b262cb726c5ab2a360f79884d008fba1"
server_key_iv: "f0a28583e65aa1094506993a10b9effd"
server_key_version: "encryptionkey@0"
I20260810 23:37:21.278795 10130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "567923278e4b471fabaa9b0aa14b05af"
format_stamp: "Formatted at 2026-08-10 23:37:21 on dist-test-slave-04bb"
server_key: "b262cb726c5ab2a360f79884d008fba1"
server_key_iv: "f0a28583e65aa1094506993a10b9effd"
server_key_version: "encryptionkey@0"
I20260810 23:37:21.284389 10130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 23:37:21.288123 10153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:21.289486 10130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:37:21.289673 10130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "567923278e4b471fabaa9b0aa14b05af"
format_stamp: "Formatted at 2026-08-10 23:37:21 on dist-test-slave-04bb"
server_key: "b262cb726c5ab2a360f79884d008fba1"
server_key_iv: "f0a28583e65aa1094506993a10b9effd"
server_key_version: "encryptionkey@0"
I20260810 23:37:21.289893 10130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:21.296005 10130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:21.300307 10130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:21.300683 10130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:21.320361 10130 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34949
I20260810 23:37:21.320451 10228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34949 every 8 connection(s)
I20260810 23:37:21.321875 10130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:21.325912 10230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:21.332015 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 10130
I20260810 23:37:21.332230 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:21.332849 9411 external_mini_cluster.cc:1428] Setting key 9848e158467098894addb2aefa22d18b
I20260810 23:37:21.339105 10230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af: Bootstrap starting.
I20260810 23:37:21.341322 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:21.334991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42345 (local address 127.0.0.1:34949)
0810 23:37:21.335870 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:21.335885 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:21.335930 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:37:21.337737 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:21.337761 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:21.338096 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:21.338491 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:21.339652 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:21.339658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:21.339671 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:21.339687 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:21.339748 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:21.339789 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:21.340012 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:21.340145 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 23:37:21.340339 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":385,"threads_started":1}
I20260810 23:37:21.342702 10230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:21.343997 10230 log.cc:824] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:21.345207 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:21.347059 10230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af: No bootstrap required, opened a new log
I20260810 23:37:21.352764 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:21.349757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57521 (local address 127.0.0.1:34949)
0810 23:37:21.349977 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:21.349992 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:21.350448 (+ 456us) server_negotiation.cc:407] Connection header received
0810 23:37:21.350669 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:21.350679 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:21.350729 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:21.350858 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:21.351904 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:21.351910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:21.351921 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:21.351937 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:21.351983 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:21.351999 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:21.352151 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:21.352336 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 23:37:21.352403 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:37:21.356959 10230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "567923278e4b471fabaa9b0aa14b05af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34949 } }
I20260810 23:37:21.357370 10230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:21.357488 10230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567923278e4b471fabaa9b0aa14b05af, State: Initialized, Role: FOLLOWER
I20260810 23:37:21.358219 10230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "567923278e4b471fabaa9b0aa14b05af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34949 } }
I20260810 23:37:21.358485 10230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:21.358657 10230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:21.358862 10230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:21.361505 10230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "567923278e4b471fabaa9b0aa14b05af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34949 } }
I20260810 23:37:21.362057 10230 leader_election.cc:302] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 567923278e4b471fabaa9b0aa14b05af; no voters:
I20260810 23:37:21.362696 10230 leader_election.cc:288] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:21.362797 10242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:21.363509 10242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [term 1 LEADER]: Becoming Leader. State: Replica: 567923278e4b471fabaa9b0aa14b05af, State: Running, Role: LEADER
I20260810 23:37:21.364136 10242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "567923278e4b471fabaa9b0aa14b05af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34949 } }
I20260810 23:37:21.365130 10230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:21.368842 10243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "567923278e4b471fabaa9b0aa14b05af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "567923278e4b471fabaa9b0aa14b05af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34949 } } }
I20260810 23:37:21.369321 10243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:21.369270 10245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 567923278e4b471fabaa9b0aa14b05af. Latest consensus state: current_term: 1 leader_uuid: "567923278e4b471fabaa9b0aa14b05af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "567923278e4b471fabaa9b0aa14b05af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34949 } } }
I20260810 23:37:21.369549 10245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:21.370496 10259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:21.375022 10259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:21.383473 10259 catalog_manager.cc:1347] Generated new cluster ID: 28737b1b97f34cd98bf1b95a27e7db98
I20260810 23:37:21.383581 10259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:21.407202 10259 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:21.407764 10259 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:21.422863 10259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 567923278e4b471fabaa9b0aa14b05af: Generated new TSK 0
I20260810 23:37:21.423981 10259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:21.437109 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 10130
W20260810 23:37:21.438623 10221 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42345
W20260810 23:37:21.439344 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 10130: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:37:21.439520 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 10130
2026-08-10T23:37:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:21 dist-test-slave-04bb krb5kdc[10280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:21 dist-test-slave-04bb krb5kdc[10280](info): set up 2 sockets
Aug 10 23:37:21 dist-test-slave-04bb krb5kdc[10280](info): commencing operation
krb5kdc: starting...
W20260810 23:37:22.837904 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.351s user 0.001s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:23 dist-test-slave-04bb krb5kdc[10280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:37:23.166330 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:23.514508 10300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:23.514791 10300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:23.514983 10300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:23.522094 10300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:23.522212 10300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:23.522327 10300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:23.522433 10300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:23.522513 10300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:23.532377 10300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:23.533762 10300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:23.535156 10300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:23.547569 10305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:23.547538 10306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:23.547610 10310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:23.548955 10300 server_base.cc:1034] running on GCE node
I20260810 23:37:23.549872 10300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:23.551353 10300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:23.552634 10300 hybrid_clock.cc:648] HybridClock initialized: now 1786405043552592 us; error 54 us; skew 500 ppm
I20260810 23:37:23.553014 10300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:23 dist-test-slave-04bb krb5kdc[10280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:23.556264 10300 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:37:23.558296 10300 webserver.cc:460] Webserver started at http://127.0.0.1:35601/ using document root <none> and password file <none>
I20260810 23:37:23.559037 10300 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:23.559208 10300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:23.559542 10300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:23.562690 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "e59095400ed4476caee3b0ac123c9920"
format_stamp: "Formatted at 2026-08-10 23:37:23 on dist-test-slave-04bb"
server_key: "d02c4a4ca9697b216c6d04107891b445"
server_key_iv: "75cd60992eab452a041bccae08442757"
server_key_version: "encryptionkey@0"
I20260810 23:37:23.563477 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "e59095400ed4476caee3b0ac123c9920"
format_stamp: "Formatted at 2026-08-10 23:37:23 on dist-test-slave-04bb"
server_key: "d02c4a4ca9697b216c6d04107891b445"
server_key_iv: "75cd60992eab452a041bccae08442757"
server_key_version: "encryptionkey@0"
I20260810 23:37:23.568919 10300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260810 23:37:23.572675 10321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:23.574092 10300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:37:23.574324 10300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "e59095400ed4476caee3b0ac123c9920"
format_stamp: "Formatted at 2026-08-10 23:37:23 on dist-test-slave-04bb"
server_key: "d02c4a4ca9697b216c6d04107891b445"
server_key_iv: "75cd60992eab452a041bccae08442757"
server_key_version: "encryptionkey@0"
I20260810 23:37:23.574540 10300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:23.600026 10300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:23.605094 10300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:23.605427 10300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:23.625133 10300 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33569
I20260810 23:37:23.625222 10397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33569 every 8 connection(s)
I20260810 23:37:23.626533 10300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:23.630184 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 10300
I20260810 23:37:23.630398 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:23.630380 10398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:23.630885 9411 external_mini_cluster.cc:1428] Setting key fa0660668343510b46472e3a52bb9e6f
I20260810 23:37:23.645339 10398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920: Bootstrap starting.
Aug 10 23:37:23 dist-test-slave-04bb krb5kdc[10280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:37:23.649362 10398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:23.650786 10398 log.cc:824] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:23.654558 10398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920: No bootstrap required, opened a new log
I20260810 23:37:23.656270 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:23.632728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45307 (local address 127.0.0.1:33569)
0810 23:37:23.633666 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:23.633688 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:23.633745 (+ 57us) server_negotiation.cc:407] Connection header received
0810 23:37:23.635688 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:23.635717 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:23.636279 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:23.636750 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:23.638056 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:23.640046 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:23.641485 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:23.642608 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:23.642810 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:23.647009 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:23.647019 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:23.647040 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:23.647095 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:23.651243 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:23.652267 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:23.652275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:23.652281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:23.652558 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:23.652949 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:23.652954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:23.652958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:23.653669 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:23.653985 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:23.654415 (+ 430us) server_negotiation.cc:299] Negotiation successful
0810 23:37:23.654867 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":364,"threads_started":1}
I20260810 23:37:23.660341 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:23.663594 10398 raft_consensus.cc:348] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59095400ed4476caee3b0ac123c9920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33569 } }
I20260810 23:37:23.663957 10398 raft_consensus.cc:374] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:23.664058 10398 raft_consensus.cc:729] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59095400ed4476caee3b0ac123c9920, State: Initialized, Role: FOLLOWER
I20260810 23:37:23.664808 10398 consensus_queue.cc:260] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59095400ed4476caee3b0ac123c9920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33569 } }
I20260810 23:37:23.665161 10398 raft_consensus.cc:388] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:23.665359 10398 raft_consensus.cc:482] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:23.665542 10398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:23.668213 10398 raft_consensus.cc:504] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59095400ed4476caee3b0ac123c9920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33569 } }
I20260810 23:37:23.668833 10398 leader_election.cc:302] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e59095400ed4476caee3b0ac123c9920; no voters:
I20260810 23:37:23.669430 10398 leader_election.cc:288] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:23.669579 10413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:23.670248 10413 raft_consensus.cc:686] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [term 1 LEADER]: Becoming Leader. State: Replica: e59095400ed4476caee3b0ac123c9920, State: Running, Role: LEADER
I20260810 23:37:23.670889 10413 consensus_queue.cc:237] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59095400ed4476caee3b0ac123c9920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33569 } }
I20260810 23:37:23.674290 10416 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e59095400ed4476caee3b0ac123c9920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59095400ed4476caee3b0ac123c9920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33569 } } }
I20260810 23:37:23.674687 10413 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e59095400ed4476caee3b0ac123c9920. Latest consensus state: current_term: 1 leader_uuid: "e59095400ed4476caee3b0ac123c9920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59095400ed4476caee3b0ac123c9920" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33569 } } }
I20260810 23:37:23.674806 10416 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:23.674899 10413 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:23.677237 10398 sys_catalog.cc:564] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:23.680154 10418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:23.682696 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:23.664974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39425 (local address 127.0.0.1:33569)
0810 23:37:23.665167 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:23.665175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:23.665694 (+ 519us) server_negotiation.cc:407] Connection header received
0810 23:37:23.665927 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:23.665932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:23.666013 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:23.666142 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:23.667403 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:23.668440 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:23.670316 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:23.671048 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:23.671168 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:23.674858 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:23.674867 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:23.674875 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:23.674923 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:23.679297 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:23.680236 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:23.680244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:23.680249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:23.680409 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:23.680923 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:23.680927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:23.680929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:23.681269 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:23.681454 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:23.682036 (+ 582us) server_negotiation.cc:299] Negotiation successful
0810 23:37:23.682268 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:37:23.688905 10418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:23.697661 10418 catalog_manager.cc:1347] Generated new cluster ID: fabc8dd519324b3687cf228bc1b1d819
I20260810 23:37:23.697783 10418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:23.707116 10418 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:23.708572 10418 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:23.718459 10418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e59095400ed4476caee3b0ac123c9920: Generated new TSK 0
I20260810 23:37:23.719451 10418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:23.731302 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 10300
I20260810 23:37:23.732416 10391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:23.822592 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 10300
2026-08-10T23:37:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:23 dist-test-slave-04bb krb5kdc[10449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:23 dist-test-slave-04bb krb5kdc[10449](info): set up 2 sockets
Aug 10 23:37:23 dist-test-slave-04bb krb5kdc[10449](info): commencing operation
krb5kdc: starting...
W20260810 23:37:25.192958 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.325s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:25 dist-test-slave-04bb krb5kdc[10449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:37:25.523447 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38761
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:25.869928 10471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:25.870285 10471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:25.870455 10471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:25.877645 10471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:25.877786 10471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:25.877895 10471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:25.878016 10471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:25.878095 10471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:25.888059 10471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:25.889449 10471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:25.890836 10471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:25.903173 10482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:25.903123 10484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:25.903205 10481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:25.904177 10471 server_base.cc:1034] running on GCE node
I20260810 23:37:25.905046 10471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:25.906507 10471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:25.907712 10471 hybrid_clock.cc:648] HybridClock initialized: now 1786405045907662 us; error 47 us; skew 500 ppm
I20260810 23:37:25.908094 10471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:25 dist-test-slave-04bb krb5kdc[10449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:25.916347 10471 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:37:25.918025 10471 webserver.cc:460] Webserver started at http://127.0.0.1:36415/ using document root <none> and password file <none>
I20260810 23:37:25.918711 10471 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:25.918814 10471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:25.919081 10471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:25.922142 10471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "3d2ea21bd43048e68804005dbc2ef4a5"
format_stamp: "Formatted at 2026-08-10 23:37:25 on dist-test-slave-04bb"
server_key: "627ba2cba0aefc00aa5b53975b790262"
server_key_iv: "1e2084d366a81f6f24be55de6e80e4d9"
server_key_version: "encryptionkey@0"
I20260810 23:37:25.922861 10471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "3d2ea21bd43048e68804005dbc2ef4a5"
format_stamp: "Formatted at 2026-08-10 23:37:25 on dist-test-slave-04bb"
server_key: "627ba2cba0aefc00aa5b53975b790262"
server_key_iv: "1e2084d366a81f6f24be55de6e80e4d9"
server_key_version: "encryptionkey@0"
I20260810 23:37:25.929143 10471 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 23:37:25.932754 10494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:25.934237 10471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:37:25.934456 10471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "3d2ea21bd43048e68804005dbc2ef4a5"
format_stamp: "Formatted at 2026-08-10 23:37:25 on dist-test-slave-04bb"
server_key: "627ba2cba0aefc00aa5b53975b790262"
server_key_iv: "1e2084d366a81f6f24be55de6e80e4d9"
server_key_version: "encryptionkey@0"
I20260810 23:37:25.934652 10471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:25.956658 10471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:25.961758 10471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:25.962090 10471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:25.983309 10471 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38761
I20260810 23:37:25.983349 10569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38761 every 8 connection(s)
I20260810 23:37:25.984869 10471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:25.986078 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 10471
I20260810 23:37:25.986284 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:25.986869 9411 external_mini_cluster.cc:1428] Setting key 485188e18a84d62a807179bd71532848
I20260810 23:37:25.990017 10571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:26.002882 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:25.989060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35629 (local address 127.0.0.1:38761)
0810 23:37:25.989913 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:25.989930 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:25.989989 (+ 59us) server_negotiation.cc:407] Connection header received
0810 23:37:25.992409 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:25.992436 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:25.992956 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:25.993454 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:25.993982 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:25.996026 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:25.997479 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:25.998487 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:25.998684 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:25.999865 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:25.999876 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:25.999899 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:26.000694 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:26.000800 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:26.000832 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:26.000966 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:26.001211 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 23:37:26.001503 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":375,"threads_started":1}
I20260810 23:37:26.004421 10571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5: Bootstrap starting.
I20260810 23:37:26.007076 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:26.008004 10571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:26.009114 10571 log.cc:824] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:26.012465 10571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5: No bootstrap required, opened a new log
I20260810 23:37:26.019016 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:26.011855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38071 (local address 127.0.0.1:38761)
0810 23:37:26.012076 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:26.012085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:26.012640 (+ 555us) server_negotiation.cc:407] Connection header received
0810 23:37:26.012858 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:26.012863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:26.012953 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:26.013080 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:26.013638 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:26.014745 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:26.016153 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:26.016925 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:26.017048 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:26.018210 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:26.018217 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:26.018222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:26.018247 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:26.018301 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:26.018327 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:26.018424 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:26.018560 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 23:37:26.018663 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:37:26.021942 10571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2ea21bd43048e68804005dbc2ef4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } }
I20260810 23:37:26.022431 10571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:26.022572 10571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d2ea21bd43048e68804005dbc2ef4a5, State: Initialized, Role: FOLLOWER
I20260810 23:37:26.023231 10571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2ea21bd43048e68804005dbc2ef4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } }
I20260810 23:37:26.023442 10571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:26.023547 10571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:26.023684 10571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:26.025875 10571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2ea21bd43048e68804005dbc2ef4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } }
I20260810 23:37:26.026336 10571 leader_election.cc:302] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d2ea21bd43048e68804005dbc2ef4a5; no voters:
I20260810 23:37:26.026896 10571 leader_election.cc:288] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:26.027102 10587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:26.027947 10587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [term 1 LEADER]: Becoming Leader. State: Replica: 3d2ea21bd43048e68804005dbc2ef4a5, State: Running, Role: LEADER
I20260810 23:37:26.028735 10587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2ea21bd43048e68804005dbc2ef4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } }
I20260810 23:37:26.029394 10571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:26.032346 10589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d2ea21bd43048e68804005dbc2ef4a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2ea21bd43048e68804005dbc2ef4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } } }
I20260810 23:37:26.032596 10591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d2ea21bd43048e68804005dbc2ef4a5. Latest consensus state: current_term: 1 leader_uuid: "3d2ea21bd43048e68804005dbc2ef4a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2ea21bd43048e68804005dbc2ef4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38761 } } }
I20260810 23:37:26.032918 10589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:26.032918 10591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:26.035071 10601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:26.041244 10601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:26.051295 10601 catalog_manager.cc:1347] Generated new cluster ID: db50b52b1bb64e7b82e99c9231ad73ab
I20260810 23:37:26.051432 10601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:26.086253 10601 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:26.088323 10601 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:26.097673 10601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d2ea21bd43048e68804005dbc2ef4a5: Generated new TSK 0
I20260810 23:37:26.098721 10601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:26.116067 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 10471
W20260810 23:37:26.117367 10562 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35629
W20260810 23:37:26.118172 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 10471: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:37:26.118332 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 10471
2026-08-10T23:37:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:26 dist-test-slave-04bb krb5kdc[10624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:26 dist-test-slave-04bb krb5kdc[10624](info): set up 2 sockets
Aug 10 23:37:26 dist-test-slave-04bb krb5kdc[10624](info): commencing operation
krb5kdc: starting...
W20260810 23:37:27.498266 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.332s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:27 dist-test-slave-04bb krb5kdc[10624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:37:27.836732 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:28.194278 10643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:28.194572 10643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:28.194726 10643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:28.202088 10643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:28.202199 10643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:28.202313 10643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:28.202414 10643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:28.202494 10643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:28.212468 10643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:28.213814 10643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:28.215164 10643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:28.227070 10652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:28.227139 10655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:28.227197 10657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:28.228184 10643 server_base.cc:1034] running on GCE node
I20260810 23:37:28.229092 10643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:28.230741 10643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:28.232012 10643 hybrid_clock.cc:648] HybridClock initialized: now 1786405048231974 us; error 41 us; skew 500 ppm
I20260810 23:37:28.232457 10643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:28 dist-test-slave-04bb krb5kdc[10624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:28.235821 10643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:37:28.237421 10643 webserver.cc:460] Webserver started at http://127.0.0.1:39017/ using document root <none> and password file <none>
I20260810 23:37:28.238139 10643 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:28.238293 10643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:28.238631 10643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:28.241772 10643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "abbe208aeec44d82aa061d0b4bbe194f"
format_stamp: "Formatted at 2026-08-10 23:37:28 on dist-test-slave-04bb"
server_key: "6039593cf538c0fc5edd110e455fa022"
server_key_iv: "a9abf8043ef27839973a49f29bb7a520"
server_key_version: "encryptionkey@0"
I20260810 23:37:28.242548 10643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "abbe208aeec44d82aa061d0b4bbe194f"
format_stamp: "Formatted at 2026-08-10 23:37:28 on dist-test-slave-04bb"
server_key: "6039593cf538c0fc5edd110e455fa022"
server_key_iv: "a9abf8043ef27839973a49f29bb7a520"
server_key_version: "encryptionkey@0"
I20260810 23:37:28.247841 10643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 23:37:28.251322 10670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:28.252656 10643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:37:28.252838 10643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "abbe208aeec44d82aa061d0b4bbe194f"
format_stamp: "Formatted at 2026-08-10 23:37:28 on dist-test-slave-04bb"
server_key: "6039593cf538c0fc5edd110e455fa022"
server_key_iv: "a9abf8043ef27839973a49f29bb7a520"
server_key_version: "encryptionkey@0"
I20260810 23:37:28.253036 10643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:28.258662 10643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:28.262889 10643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:28.263200 10643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:28.282740 10643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40269
I20260810 23:37:28.282793 10752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40269 every 8 connection(s)
I20260810 23:37:28.284168 10643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:28.288051 10753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:28.291509 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 10643
I20260810 23:37:28.291690 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:28.292271 9411 external_mini_cluster.cc:1428] Setting key 4a137316df12ead674f73b246f758a08
I20260810 23:37:28.300690 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:28.294247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42499 (local address 127.0.0.1:40269)
0810 23:37:28.295048 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:28.295060 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:28.295110 (+ 50us) server_negotiation.cc:407] Connection header received
0810 23:37:28.297247 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:28.297282 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:28.297631 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:28.297956 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:28.299032 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:28.299039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:28.299051 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:28.299077 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:28.299164 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:28.299188 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:28.299366 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:28.299494 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 23:37:28.299691 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":372,"threads_started":1}
I20260810 23:37:28.301532 10753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f: Bootstrap starting.
I20260810 23:37:28.304358 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:28.305018 10753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:28.306025 10753 log.cc:824] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:28.309461 10753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f: No bootstrap required, opened a new log
I20260810 23:37:28.311215 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:28.308854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37529 (local address 127.0.0.1:40269)
0810 23:37:28.309026 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:28.309033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:28.309401 (+ 368us) server_negotiation.cc:407] Connection header received
0810 23:37:28.309501 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:28.309509 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:28.309564 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:28.309665 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:28.310570 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:28.310576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:28.310582 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:28.310599 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:28.310643 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:28.310662 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:28.310779 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:28.310848 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 23:37:28.310906 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:37:28.318123 10753 raft_consensus.cc:348] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbe208aeec44d82aa061d0b4bbe194f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } }
I20260810 23:37:28.318611 10753 raft_consensus.cc:374] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:28.318758 10753 raft_consensus.cc:729] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbe208aeec44d82aa061d0b4bbe194f, State: Initialized, Role: FOLLOWER
I20260810 23:37:28.319388 10753 consensus_queue.cc:260] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbe208aeec44d82aa061d0b4bbe194f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } }
I20260810 23:37:28.319608 10753 raft_consensus.cc:388] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:28.319711 10753 raft_consensus.cc:482] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:28.319841 10753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:28.322000 10753 raft_consensus.cc:504] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbe208aeec44d82aa061d0b4bbe194f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } }
I20260810 23:37:28.322443 10753 leader_election.cc:302] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abbe208aeec44d82aa061d0b4bbe194f; no voters:
I20260810 23:37:28.322957 10753 leader_election.cc:288] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:28.323115 10773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:28.324105 10773 raft_consensus.cc:686] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [term 1 LEADER]: Becoming Leader. State: Replica: abbe208aeec44d82aa061d0b4bbe194f, State: Running, Role: LEADER
I20260810 23:37:28.324726 10773 consensus_queue.cc:237] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbe208aeec44d82aa061d0b4bbe194f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } }
I20260810 23:37:28.325474 10753 sys_catalog.cc:564] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:28.327733 10775 sys_catalog.cc:455] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [sys.catalog]: SysCatalogTable state changed. Reason: New leader abbe208aeec44d82aa061d0b4bbe194f. Latest consensus state: current_term: 1 leader_uuid: "abbe208aeec44d82aa061d0b4bbe194f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbe208aeec44d82aa061d0b4bbe194f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } } }
I20260810 23:37:28.328269 10775 sys_catalog.cc:458] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:28.327761 10774 sys_catalog.cc:455] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abbe208aeec44d82aa061d0b4bbe194f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbe208aeec44d82aa061d0b4bbe194f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } } }
I20260810 23:37:28.329599 10774 sys_catalog.cc:458] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:28.331357 10781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:28.337113 10781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:28.345939 10781 catalog_manager.cc:1347] Generated new cluster ID: b1f0479facd24ad7a9da7c7a3c67172b
I20260810 23:37:28.346046 10781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:28.373649 10781 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:28.374193 10781 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:28.380553 10781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abbe208aeec44d82aa061d0b4bbe194f: Generated new TSK 0
I20260810 23:37:28.381577 10781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:28.389004 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 10643
W20260810 23:37:28.390213 10743 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42499
W20260810 23:37:28.390949 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 10643: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:37:28.391101 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 10643
2026-08-10T23:37:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:28 dist-test-slave-04bb krb5kdc[10800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:28 dist-test-slave-04bb krb5kdc[10800](info): set up 2 sockets
Aug 10 23:37:28 dist-test-slave-04bb krb5kdc[10800](info): commencing operation
krb5kdc: starting...
W20260810 23:37:29.778023 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.338s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:29 dist-test-slave-04bb krb5kdc[10800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:37:30.133909 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32901
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:30.477897 10820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:30.478255 10820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:30.478431 10820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:30.485585 10820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:30.485736 10820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:30.485853 10820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:30.485961 10820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:30.486053 10820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:30.495775 10820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:30.497219 10820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:30.498555 10820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:30.511051 10830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:30.511092 10833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:30.511154 10831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:30.512459 10820 server_base.cc:1034] running on GCE node
I20260810 23:37:30.513360 10820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:30.514859 10820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:30.516111 10820 hybrid_clock.cc:648] HybridClock initialized: now 1786405050516068 us; error 39 us; skew 500 ppm
I20260810 23:37:30.516479 10820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:30 dist-test-slave-04bb krb5kdc[10800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:30.519801 10820 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:37:30.521368 10820 webserver.cc:460] Webserver started at http://127.0.0.1:35531/ using document root <none> and password file <none>
I20260810 23:37:30.522063 10820 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:30.522167 10820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:30.522439 10820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:30.525557 10820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "31a48409c4024a2883c5f1342db618e2"
format_stamp: "Formatted at 2026-08-10 23:37:30 on dist-test-slave-04bb"
server_key: "d8fe1be87e24485043aa3012a3289c9a"
server_key_iv: "d4859983bba6df130c52d9b4b7a4faf3"
server_key_version: "encryptionkey@0"
I20260810 23:37:30.526320 10820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "31a48409c4024a2883c5f1342db618e2"
format_stamp: "Formatted at 2026-08-10 23:37:30 on dist-test-slave-04bb"
server_key: "d8fe1be87e24485043aa3012a3289c9a"
server_key_iv: "d4859983bba6df130c52d9b4b7a4faf3"
server_key_version: "encryptionkey@0"
I20260810 23:37:30.531744 10820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:37:30.535804 10854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:30.537351 10820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:37:30.537559 10820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "31a48409c4024a2883c5f1342db618e2"
format_stamp: "Formatted at 2026-08-10 23:37:30 on dist-test-slave-04bb"
server_key: "d8fe1be87e24485043aa3012a3289c9a"
server_key_iv: "d4859983bba6df130c52d9b4b7a4faf3"
server_key_version: "encryptionkey@0"
I20260810 23:37:30.537796 10820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:30.556213 10820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:30.560784 10820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:30.561106 10820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:30.581446 10820 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32901
I20260810 23:37:30.581478 10927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32901 every 8 connection(s)
I20260810 23:37:30.583027 10820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:30.586532 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 10820
I20260810 23:37:30.586752 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:30.587114 10928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:30.587306 9411 external_mini_cluster.cc:1428] Setting key f2d431c2540e627a69801a388902b6b0
I20260810 23:37:30.601348 10928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2: Bootstrap starting.
Aug 10 23:37:30 dist-test-slave-04bb krb5kdc[10800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:37:30.604715 10928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:30.605844 10928 log.cc:824] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:30.608896 10928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2: No bootstrap required, opened a new log
I20260810 23:37:30.611440 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:30.589382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:34749 (local address 127.0.0.1:32901)
0810 23:37:30.590315 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:30.590328 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:30.590378 (+ 50us) server_negotiation.cc:407] Connection header received
0810 23:37:30.592289 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:30.592313 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:30.592861 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:30.593239 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:30.595092 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:30.596337 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:30.598245 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:30.598986 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:30.599221 (+ 235us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:30.603695 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:30.603703 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:30.603717 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:30.603758 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:30.607104 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:30.607938 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:30.607945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:30.607951 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:30.608122 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:30.608652 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:30.608656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:30.608658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:30.609237 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:30.609635 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:30.610063 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 23:37:30.610391 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":420,"threads_started":1}
I20260810 23:37:30.615829 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:30.617681 10928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a48409c4024a2883c5f1342db618e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32901 } }
I20260810 23:37:30.618057 10928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:30.618170 10928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a48409c4024a2883c5f1342db618e2, State: Initialized, Role: FOLLOWER
I20260810 23:37:30.619216 10928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a48409c4024a2883c5f1342db618e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32901 } }
I20260810 23:37:30.619699 10939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:37:30.619964 10939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a48409c4024a2883c5f1342db618e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32901 } }
I20260810 23:37:30.620922 10939 leader_election.cc:302] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31a48409c4024a2883c5f1342db618e2; no voters:
I20260810 23:37:30.619642 10928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:30.621824 10928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:30.621861 10939 leader_election.cc:288] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260810 23:37:30.622013 10928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:30.624476 10928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a48409c4024a2883c5f1342db618e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32901 } }
I20260810 23:37:30.624795 10928 leader_election.cc:302] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31a48409c4024a2883c5f1342db618e2; no voters:
I20260810 23:37:30.624873 10947 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260810 23:37:30.625036 10928 leader_election.cc:288] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:30.625406 10939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:30.626420 10939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [term 1 LEADER]: Becoming Leader. State: Replica: 31a48409c4024a2883c5f1342db618e2, State: Running, Role: LEADER
I20260810 23:37:30.626998 10939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a48409c4024a2883c5f1342db618e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32901 } }
I20260810 23:37:30.627808 10928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:30.629256 10948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31a48409c4024a2883c5f1342db618e2. Latest consensus state: current_term: 1 leader_uuid: "31a48409c4024a2883c5f1342db618e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a48409c4024a2883c5f1342db618e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32901 } } }
I20260810 23:37:30.629309 10947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31a48409c4024a2883c5f1342db618e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31a48409c4024a2883c5f1342db618e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32901 } } }
I20260810 23:37:30.629719 10948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:30.629769 10947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:30.630815 10957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:30.640270 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:30.620612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:60211 (local address 127.0.0.1:32901)
0810 23:37:30.620780 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:30.620788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:30.621351 (+ 563us) server_negotiation.cc:407] Connection header received
0810 23:37:30.621565 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:30.621572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:30.621690 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:30.621819 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:30.623562 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:30.624321 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:30.626383 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:30.626952 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:30.627035 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:30.630637 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:30.630656 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:30.630665 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:30.630712 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:30.634916 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:30.636189 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:30.636198 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:30.636205 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:30.636387 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:30.637157 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:30.637165 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:30.637171 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:30.637673 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:30.638072 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:30.639511 (+ 1439us) server_negotiation.cc:299] Negotiation successful
0810 23:37:30.639758 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:37:30.649541 10957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:30.659983 10957 catalog_manager.cc:1347] Generated new cluster ID: 7d54ae809f334bb09734b5485e7a8105
I20260810 23:37:30.660136 10957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:30.688784 10957 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:30.690160 10957 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:30.702263 10957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31a48409c4024a2883c5f1342db618e2: Generated new TSK 0
I20260810 23:37:30.703357 10957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:30.723450 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 10820
I20260810 23:37:30.724992 10923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:30.820102 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 10820
W20260810 23:37:30.850095 10798 connection.cc:535] client connection to 127.0.0.1:32901 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32901): Connection reset by peer (error 104)
2026-08-10T23:37:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:30 dist-test-slave-04bb krb5kdc[10980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:30 dist-test-slave-04bb krb5kdc[10980](info): set up 2 sockets
Aug 10 23:37:30 dist-test-slave-04bb krb5kdc[10980](info): commencing operation
krb5kdc: starting...
W20260810 23:37:32.225922 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.355s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:32 dist-test-slave-04bb krb5kdc[10980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:37:32.572063 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:32.920399 11003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:32.920712 11003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:32.920809 11003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:32.927877 11003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:32.927994 11003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:32.928042 11003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:32.928105 11003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:32.928150 11003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:32.937984 11003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:32.939322 11003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:32.940634 11003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:32.953418 11017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:32.953440 11020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:32.953554 11015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:32.955494 11003 server_base.cc:1034] running on GCE node
I20260810 23:37:32.956429 11003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:32.957919 11003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:32.959136 11003 hybrid_clock.cc:648] HybridClock initialized: now 1786405052959087 us; error 41 us; skew 500 ppm
I20260810 23:37:32.959519 11003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:32 dist-test-slave-04bb krb5kdc[10980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:32.962836 11003 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:37:32.964402 11003 webserver.cc:460] Webserver started at http://127.0.0.1:43067/ using document root <none> and password file <none>
I20260810 23:37:32.965270 11003 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:32.965422 11003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:32.965744 11003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:32.968778 11003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "7ad805f89743459480a109c4dcaf1764"
format_stamp: "Formatted at 2026-08-10 23:37:32 on dist-test-slave-04bb"
server_key: "872eba9b20dd4c5755d438b109d37168"
server_key_iv: "a25f84dc98f7b4546aec572367a63fd8"
server_key_version: "encryptionkey@0"
I20260810 23:37:32.969578 11003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "7ad805f89743459480a109c4dcaf1764"
format_stamp: "Formatted at 2026-08-10 23:37:32 on dist-test-slave-04bb"
server_key: "872eba9b20dd4c5755d438b109d37168"
server_key_iv: "a25f84dc98f7b4546aec572367a63fd8"
server_key_version: "encryptionkey@0"
I20260810 23:37:32.974858 11003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:37:32.978541 11033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:32.979959 11003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:37:32.980163 11003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "7ad805f89743459480a109c4dcaf1764"
format_stamp: "Formatted at 2026-08-10 23:37:32 on dist-test-slave-04bb"
server_key: "872eba9b20dd4c5755d438b109d37168"
server_key_iv: "a25f84dc98f7b4546aec572367a63fd8"
server_key_version: "encryptionkey@0"
I20260810 23:37:32.980409 11003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:33.006620 11003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:33.011343 11003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:33.011656 11003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:33.030155 11003 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46183
I20260810 23:37:33.030197 11104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46183 every 8 connection(s)
I20260810 23:37:33.031555 11003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:33.035343 11105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:33.036341 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 11003
I20260810 23:37:33.036572 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:33.037051 9411 external_mini_cluster.cc:1428] Setting key ad0490b10af7667d7ffe129b23f95b42
I20260810 23:37:33.050444 11105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764: Bootstrap starting.
I20260810 23:37:33.052618 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:33.038897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:54815 (local address 127.0.0.1:46183)
0810 23:37:33.039681 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:33.039697 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:33.039767 (+ 70us) server_negotiation.cc:407] Connection header received
0810 23:37:33.042206 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:33.042234 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:33.042758 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:33.043203 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:33.043696 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:33.045570 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:33.047039 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:33.048143 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:33.048313 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:33.049624 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:33.049636 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:33.049655 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:33.050387 (+ 732us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:33.050494 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:33.050532 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:33.050804 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:33.051063 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 23:37:33.051242 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":335,"threads_started":1}
I20260810 23:37:33.054593 11105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:33.055827 11105 log.cc:824] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:33.056939 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:33.059217 11105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764: No bootstrap required, opened a new log
I20260810 23:37:33.069698 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:33.062481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:37257 (local address 127.0.0.1:46183)
0810 23:37:33.062713 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:33.062720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:33.063297 (+ 577us) server_negotiation.cc:407] Connection header received
0810 23:37:33.063436 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:33.063441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:33.063523 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:33.063656 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:33.064157 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:33.065356 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:33.066815 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:33.067567 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:33.067704 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:33.068867 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:33.068875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:33.068880 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:33.068902 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:33.068964 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:33.068984 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:33.069105 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:33.069309 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 23:37:33.069362 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:37:33.069707 11105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad805f89743459480a109c4dcaf1764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46183 } }
I20260810 23:37:33.070211 11105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:33.070358 11105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad805f89743459480a109c4dcaf1764, State: Initialized, Role: FOLLOWER
I20260810 23:37:33.070971 11105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad805f89743459480a109c4dcaf1764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46183 } }
I20260810 23:37:33.071234 11105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:33.071416 11105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:33.071604 11105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:33.073915 11105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad805f89743459480a109c4dcaf1764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46183 } }
I20260810 23:37:33.074424 11105 leader_election.cc:302] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ad805f89743459480a109c4dcaf1764; no voters:
I20260810 23:37:33.075058 11105 leader_election.cc:288] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:33.075286 11121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:33.076491 11121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [term 1 LEADER]: Becoming Leader. State: Replica: 7ad805f89743459480a109c4dcaf1764, State: Running, Role: LEADER
I20260810 23:37:33.077056 11121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad805f89743459480a109c4dcaf1764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46183 } }
I20260810 23:37:33.077499 11105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:33.080090 11123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ad805f89743459480a109c4dcaf1764. Latest consensus state: current_term: 1 leader_uuid: "7ad805f89743459480a109c4dcaf1764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad805f89743459480a109c4dcaf1764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46183 } } }
I20260810 23:37:33.080186 11122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ad805f89743459480a109c4dcaf1764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad805f89743459480a109c4dcaf1764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46183 } } }
I20260810 23:37:33.080572 11123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:33.080572 11122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:33.081475 11135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:33.092002 11135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:33.101284 11135 catalog_manager.cc:1347] Generated new cluster ID: f417e5b01a37470daf3674336556be1d
I20260810 23:37:33.101389 11135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:33.125276 11135 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:33.126557 11135 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:33.136442 11135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ad805f89743459480a109c4dcaf1764: Generated new TSK 0
I20260810 23:37:33.137375 11135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:33.143990 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11003
W20260810 23:37:33.145737 11100 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.220:54815
W20260810 23:37:33.146523 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11003: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:37:33.146690 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 11003
W20260810 23:37:33.176741 10977 connection.cc:535] client connection to 127.0.0.1:46183 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46183): Connection reset by peer (error 104)
2026-08-10T23:37:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:33 dist-test-slave-04bb krb5kdc[11158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:33 dist-test-slave-04bb krb5kdc[11158](info): set up 2 sockets
Aug 10 23:37:33 dist-test-slave-04bb krb5kdc[11158](info): commencing operation
krb5kdc: starting...
W20260810 23:37:34.530033 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.334s user 0.002s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:34 dist-test-slave-04bb krb5kdc[11158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:37:34.880795 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:35.237277 11177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:35.237571 11177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:35.237671 11177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:35.244946 11177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:35.245054 11177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:35.245101 11177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:35.245172 11177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:35.245210 11177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:35.254868 11177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:35.256078 11177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:35.257381 11177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:35.269408 11187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:35.269506 11183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:35.269448 11184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:35.270578 11177 server_base.cc:1034] running on GCE node
I20260810 23:37:35.271338 11177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:35.272675 11177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:35.273888 11177 hybrid_clock.cc:648] HybridClock initialized: now 1786405055273852 us; error 29 us; skew 500 ppm
I20260810 23:37:35.274246 11177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:35 dist-test-slave-04bb krb5kdc[11158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:35.277693 11177 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:37:35.279237 11177 webserver.cc:460] Webserver started at http://127.0.0.1:35017/ using document root <none> and password file <none>
I20260810 23:37:35.279944 11177 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:35.280095 11177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:35.280433 11177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:35.283555 11177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "297bd1c168fe44e3914c0051f1208a62"
format_stamp: "Formatted at 2026-08-10 23:37:35 on dist-test-slave-04bb"
server_key: "6edcdc08939d8633a653573c223cd2a6"
server_key_iv: "f9ac61da87620415b0a95082840f50fe"
server_key_version: "encryptionkey@0"
I20260810 23:37:35.284340 11177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "297bd1c168fe44e3914c0051f1208a62"
format_stamp: "Formatted at 2026-08-10 23:37:35 on dist-test-slave-04bb"
server_key: "6edcdc08939d8633a653573c223cd2a6"
server_key_iv: "f9ac61da87620415b0a95082840f50fe"
server_key_version: "encryptionkey@0"
I20260810 23:37:35.289691 11177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260810 23:37:35.293332 11197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:35.294692 11177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:37:35.294878 11177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "297bd1c168fe44e3914c0051f1208a62"
format_stamp: "Formatted at 2026-08-10 23:37:35 on dist-test-slave-04bb"
server_key: "6edcdc08939d8633a653573c223cd2a6"
server_key_iv: "f9ac61da87620415b0a95082840f50fe"
server_key_version: "encryptionkey@0"
I20260810 23:37:35.295079 11177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:35.301098 11177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:35.305518 11177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:35.305832 11177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:35.325322 11177 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35917
I20260810 23:37:35.325405 11276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35917 every 8 connection(s)
I20260810 23:37:35.326717 11177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:35.330591 11277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:35.336437 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 11177
I20260810 23:37:35.336650 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:35.337275 9411 external_mini_cluster.cc:1428] Setting key 44f6f622b9b7ac198c797d160816f88c
I20260810 23:37:35.342907 11277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62: Bootstrap starting.
I20260810 23:37:35.346139 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:35.339454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:49167 (local address 127.0.0.1:35917)
0810 23:37:35.340428 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:35.340441 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:35.340492 (+ 51us) server_negotiation.cc:407] Connection header received
0810 23:37:35.342410 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:35.342434 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:35.342793 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:35.343229 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:35.344390 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:35.344397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:35.344415 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:35.344431 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:35.344495 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:35.344573 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:35.344857 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:35.345021 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 23:37:35.345233 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":378,"threads_started":1}
I20260810 23:37:35.346728 11277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:35.347906 11277 log.cc:824] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:35.350139 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:35.350879 11277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62: No bootstrap required, opened a new log
I20260810 23:37:35.357933 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:35.355028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:46053 (local address 127.0.0.1:35917)
0810 23:37:35.355272 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:35.355278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:35.355641 (+ 363us) server_negotiation.cc:407] Connection header received
0810 23:37:35.355835 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:35.355839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:35.355891 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:35.355976 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:35.357171 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:35.357175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:35.357179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:35.357200 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:35.357240 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:35.357256 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:35.357372 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:35.357514 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 23:37:35.357573 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260810 23:37:35.360780 11277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297bd1c168fe44e3914c0051f1208a62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35917 } }
I20260810 23:37:35.361204 11277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:35.361325 11277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297bd1c168fe44e3914c0051f1208a62, State: Initialized, Role: FOLLOWER
I20260810 23:37:35.361979 11277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297bd1c168fe44e3914c0051f1208a62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35917 } }
I20260810 23:37:35.362340 11277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:35.362520 11277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:35.362737 11277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:35.365206 11277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297bd1c168fe44e3914c0051f1208a62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35917 } }
I20260810 23:37:35.365736 11277 leader_election.cc:302] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 297bd1c168fe44e3914c0051f1208a62; no voters:
I20260810 23:37:35.366422 11277 leader_election.cc:288] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:35.366555 11296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:35.368146 11296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [term 1 LEADER]: Becoming Leader. State: Replica: 297bd1c168fe44e3914c0051f1208a62, State: Running, Role: LEADER
I20260810 23:37:35.368808 11296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297bd1c168fe44e3914c0051f1208a62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35917 } }
I20260810 23:37:35.369012 11277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:35.371743 11299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 297bd1c168fe44e3914c0051f1208a62. Latest consensus state: current_term: 1 leader_uuid: "297bd1c168fe44e3914c0051f1208a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297bd1c168fe44e3914c0051f1208a62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35917 } } }
I20260810 23:37:35.371842 11297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "297bd1c168fe44e3914c0051f1208a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "297bd1c168fe44e3914c0051f1208a62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35917 } } }
I20260810 23:37:35.372172 11299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:35.372263 11297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:35.373509 11311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:35.384155 11311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:35.393787 11311 catalog_manager.cc:1347] Generated new cluster ID: 52dfd7f2273a4e99be4e5829e5521ce2
I20260810 23:37:35.393949 11311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:35.409971 11311 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:35.410432 11311 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:35.424037 11311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 297bd1c168fe44e3914c0051f1208a62: Generated new TSK 0
I20260810 23:37:35.425122 11311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:35.440941 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11177
W20260810 23:37:35.442145 11268 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.220:49167
W20260810 23:37:35.442936 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11177: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:37:35.443105 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 11177
2026-08-10T23:37:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:35 dist-test-slave-04bb krb5kdc[11327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:35 dist-test-slave-04bb krb5kdc[11327](info): set up 2 sockets
Aug 10 23:37:35 dist-test-slave-04bb krb5kdc[11327](info): commencing operation
krb5kdc: starting...
W20260810 23:37:36.808624 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.318s user 0.001s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:37 dist-test-slave-04bb krb5kdc[11327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:37:37.160845 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:37.506889 11340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:37.507251 11340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:37.507428 11340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:37.514709 11340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:37.514865 11340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:37.514977 11340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:37.515090 11340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:37.515187 11340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:37.525182 11340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:37.526559 11340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:37.527945 11340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:37.540216 11347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:37.540071 11348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:37.540259 11353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:37.541548 11340 server_base.cc:1034] running on GCE node
I20260810 23:37:37.542361 11340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:37.543828 11340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:37.545074 11340 hybrid_clock.cc:648] HybridClock initialized: now 1786405057545013 us; error 60 us; skew 500 ppm
I20260810 23:37:37.545501 11340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:37 dist-test-slave-04bb krb5kdc[11327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:37.552000 11340 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:37:37.553599 11340 webserver.cc:460] Webserver started at http://127.0.0.1:33047/ using document root <none> and password file <none>
I20260810 23:37:37.554299 11340 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:37.554427 11340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:37.554697 11340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:37.557963 11340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "ed65272ce69741f49f8e749af3927e0e"
format_stamp: "Formatted at 2026-08-10 23:37:37 on dist-test-slave-04bb"
server_key: "2d5e917a9439b720b314b31634eee205"
server_key_iv: "5fe3970d8c7e19ce310afd1abb88bb0f"
server_key_version: "encryptionkey@0"
I20260810 23:37:37.558735 11340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "ed65272ce69741f49f8e749af3927e0e"
format_stamp: "Formatted at 2026-08-10 23:37:37 on dist-test-slave-04bb"
server_key: "2d5e917a9439b720b314b31634eee205"
server_key_iv: "5fe3970d8c7e19ce310afd1abb88bb0f"
server_key_version: "encryptionkey@0"
I20260810 23:37:37.564193 11340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:37:37.567811 11366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:37.569358 11340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:37:37.569573 11340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "ed65272ce69741f49f8e749af3927e0e"
format_stamp: "Formatted at 2026-08-10 23:37:37 on dist-test-slave-04bb"
server_key: "2d5e917a9439b720b314b31634eee205"
server_key_iv: "5fe3970d8c7e19ce310afd1abb88bb0f"
server_key_version: "encryptionkey@0"
I20260810 23:37:37.569798 11340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:37.609721 11340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:37.614274 11340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:37.614586 11340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:37.634361 11340 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39833
I20260810 23:37:37.634446 11440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39833 every 8 connection(s)
I20260810 23:37:37.635820 11340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:37.639614 11442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:37.642210 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 11340
I20260810 23:37:37.642395 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:37.642922 9411 external_mini_cluster.cc:1428] Setting key 0774bb50be139d0a993e993c1ec4c82f
I20260810 23:37:37.651744 11442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e: Bootstrap starting.
I20260810 23:37:37.654903 11442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:37.655894 11442 log.cc:824] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:37:37 dist-test-slave-04bb krb5kdc[11327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:37:37.659336 11442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e: No bootstrap required, opened a new log
I20260810 23:37:37.666621 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:37.644901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:50943 (local address 127.0.0.1:39833)
0810 23:37:37.645674 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:37.645695 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:37.645740 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:37:37.647938 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:37.647961 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:37.648358 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:37.648784 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:37.650285 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:37.651575 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:37.652991 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:37.653684 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:37.653876 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:37.658837 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:37.658844 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:37.658859 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:37.658896 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:37.662135 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:37.663168 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:37.663178 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:37.663186 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:37.663351 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:37.663903 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:37.663907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:37.663909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:37.664396 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:37.664744 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:37.665251 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 23:37:37.665563 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":342,"threads_started":1}
I20260810 23:37:37.669615 11442 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed65272ce69741f49f8e749af3927e0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39833 } }
I20260810 23:37:37.670092 11442 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:37.670240 11442 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed65272ce69741f49f8e749af3927e0e, State: Initialized, Role: FOLLOWER
I20260810 23:37:37.670943 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:37.670938 11442 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed65272ce69741f49f8e749af3927e0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39833 } }
I20260810 23:37:37.671255 11442 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:37.671393 11442 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:37.671568 11442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:37.675278 11442 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed65272ce69741f49f8e749af3927e0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39833 } }
I20260810 23:37:37.675958 11442 leader_election.cc:302] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed65272ce69741f49f8e749af3927e0e; no voters:
I20260810 23:37:37.676739 11442 leader_election.cc:288] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:37.676873 11453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:37.679136 11442 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:37.684288 11453 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [term 1 LEADER]: Becoming Leader. State: Replica: ed65272ce69741f49f8e749af3927e0e, State: Running, Role: LEADER
I20260810 23:37:37.685073 11453 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed65272ce69741f49f8e749af3927e0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39833 } }
I20260810 23:37:37.695559 11456 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed65272ce69741f49f8e749af3927e0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed65272ce69741f49f8e749af3927e0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39833 } } }
I20260810 23:37:37.696041 11456 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:37.696511 11460 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed65272ce69741f49f8e749af3927e0e. Latest consensus state: current_term: 1 leader_uuid: "ed65272ce69741f49f8e749af3927e0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed65272ce69741f49f8e749af3927e0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39833 } } }
I20260810 23:37:37.697036 11460 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e [sys.catalog]: This master's current role is: LEADER
W20260810 23:37:37.703794 11476 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:37:37.703969 11476 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:37:37.704149 11479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:37.706525 11479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:37.708643 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:37.677056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:40593 (local address 127.0.0.1:39833)
0810 23:37:37.677289 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:37.677302 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:37.683649 (+ 6347us) server_negotiation.cc:407] Connection header received
0810 23:37:37.683813 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:37.683826 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:37.683936 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:37.684055 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:37.686313 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:37.687918 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:37.690207 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:37.696508 (+ 6301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:37.696658 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:37.700419 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:37.700425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:37.700430 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:37.700467 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:37.704831 (+ 4364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:37.705923 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:37.705928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:37.705934 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:37.706079 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:37.706683 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:37.706694 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:37.706697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:37.707150 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:37.707494 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:37.708031 (+ 537us) server_negotiation.cc:299] Negotiation successful
0810 23:37:37.708240 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:37:37.716033 11479 catalog_manager.cc:1347] Generated new cluster ID: b76d99a2aab64e539083d07f8655cc21
I20260810 23:37:37.716216 11479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:37.753124 11479 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:37.754557 11479 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:37.763022 11479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed65272ce69741f49f8e749af3927e0e: Generated new TSK 0
I20260810 23:37:37.763832 11479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:37.769541 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11340
I20260810 23:37:37.771124 11431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:37.865075 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 11340
W20260810 23:37:37.893945 11323 connection.cc:535] client connection to 127.0.0.1:39833 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39833): Connection reset by peer (error 104)
2026-08-10T23:37:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:37 dist-test-slave-04bb krb5kdc[11490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:37 dist-test-slave-04bb krb5kdc[11490](info): set up 2 sockets
Aug 10 23:37:37 dist-test-slave-04bb krb5kdc[11490](info): commencing operation
krb5kdc: starting...
W20260810 23:37:39.238035 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.325s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:39 dist-test-slave-04bb krb5kdc[11490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:37:39.568435 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:39.935573 11512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:39.935873 11512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:39.936036 11512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:39.943311 11512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:39.943421 11512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:39.943506 11512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:39.943611 11512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:39.943696 11512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:39.953694 11512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:39.955039 11512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:39.956427 11512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:39.968781 11519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:39.968698 11524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:39.968730 11520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:39.970008 11512 server_base.cc:1034] running on GCE node
I20260810 23:37:39.970883 11512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:39.972397 11512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:39.973655 11512 hybrid_clock.cc:648] HybridClock initialized: now 1786405059973605 us; error 56 us; skew 500 ppm
I20260810 23:37:39.974033 11512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:39 dist-test-slave-04bb krb5kdc[11490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:39.977437 11512 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:37:39.979341 11512 webserver.cc:460] Webserver started at http://127.0.0.1:40915/ using document root <none> and password file <none>
I20260810 23:37:39.980295 11512 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:39.980407 11512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:39.980842 11512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:39.984472 11512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "601f8789a87f4c0c99820bbc2909da2b"
format_stamp: "Formatted at 2026-08-10 23:37:39 on dist-test-slave-04bb"
server_key: "9241330a0301b3bd6d56fbc45cfca485"
server_key_iv: "eb4107a1c0c8343cdff7bd94afce8b53"
server_key_version: "encryptionkey@0"
I20260810 23:37:39.985355 11512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "601f8789a87f4c0c99820bbc2909da2b"
format_stamp: "Formatted at 2026-08-10 23:37:39 on dist-test-slave-04bb"
server_key: "9241330a0301b3bd6d56fbc45cfca485"
server_key_iv: "eb4107a1c0c8343cdff7bd94afce8b53"
server_key_version: "encryptionkey@0"
I20260810 23:37:39.990722 11512 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:37:39.994305 11538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:39.995641 11512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:37:39.995854 11512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "601f8789a87f4c0c99820bbc2909da2b"
format_stamp: "Formatted at 2026-08-10 23:37:39 on dist-test-slave-04bb"
server_key: "9241330a0301b3bd6d56fbc45cfca485"
server_key_iv: "eb4107a1c0c8343cdff7bd94afce8b53"
server_key_version: "encryptionkey@0"
I20260810 23:37:39.996055 11512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:40.008818 11512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:40.013408 11512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:40.013716 11512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:40.033744 11512 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35529
I20260810 23:37:40.033769 11621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35529 every 8 connection(s)
I20260810 23:37:40.035113 11512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:40.038794 11626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:40.043910 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 11512
I20260810 23:37:40.044117 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:40.044703 9411 external_mini_cluster.cc:1428] Setting key b86b1920292b9997477cd1ee76d68eaf
I20260810 23:37:40.050333 11626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b: Bootstrap starting.
I20260810 23:37:40.053478 11626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:40.054523 11626 log.cc:824] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:40.057966 11626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b: No bootstrap required, opened a new log
I20260810 23:37:40.058597 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:40.046703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:34853 (local address 127.0.0.1:35529)
0810 23:37:40.047524 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:40.047539 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:40.047585 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:37:40.049428 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:40.049452 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:40.049874 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:40.050279 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:40.050825 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:40.052151 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:40.053579 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:40.054286 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:40.054492 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:40.055822 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:40.055830 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:40.055843 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:40.056576 (+ 733us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:40.056685 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:40.056718 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:40.056896 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:40.057142 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 23:37:40.057306 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":324,"threads_started":1}
I20260810 23:37:40.063344 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:40.067340 11626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601f8789a87f4c0c99820bbc2909da2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35529 } }
I20260810 23:37:40.067801 11626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:40.067932 11626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601f8789a87f4c0c99820bbc2909da2b, State: Initialized, Role: FOLLOWER
I20260810 23:37:40.068645 11626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601f8789a87f4c0c99820bbc2909da2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35529 } }
I20260810 23:37:40.068981 11626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:40.069136 11626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:40.069351 11626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:40.072408 11626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601f8789a87f4c0c99820bbc2909da2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35529 } }
I20260810 23:37:40.073002 11626 leader_election.cc:302] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 601f8789a87f4c0c99820bbc2909da2b; no voters:
I20260810 23:37:40.073807 11626 leader_election.cc:288] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:40.073910 11643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:40.075021 11643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [term 1 LEADER]: Becoming Leader. State: Replica: 601f8789a87f4c0c99820bbc2909da2b, State: Running, Role: LEADER
I20260810 23:37:40.075699 11643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601f8789a87f4c0c99820bbc2909da2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35529 } }
I20260810 23:37:40.076202 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:40.068909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:33135 (local address 127.0.0.1:35529)
0810 23:37:40.069092 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:40.069101 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:40.069677 (+ 576us) server_negotiation.cc:407] Connection header received
0810 23:37:40.069868 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:40.069872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:40.069957 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:40.070044 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:40.070647 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:40.071461 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:40.073308 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:40.073856 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:40.074006 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:40.075317 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:40.075324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:40.075328 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:40.075350 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:40.075411 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:40.075429 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:40.075593 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:40.075779 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 23:37:40.075840 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:37:40.076638 11626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:40.081132 11645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "601f8789a87f4c0c99820bbc2909da2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601f8789a87f4c0c99820bbc2909da2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35529 } } }
I20260810 23:37:40.081048 11644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 601f8789a87f4c0c99820bbc2909da2b. Latest consensus state: current_term: 1 leader_uuid: "601f8789a87f4c0c99820bbc2909da2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601f8789a87f4c0c99820bbc2909da2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35529 } } }
I20260810 23:37:40.081638 11645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:40.081651 11644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:40.083082 11663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:40.088346 11663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:40.099494 11663 catalog_manager.cc:1347] Generated new cluster ID: 16e4ec1db3654e2c8c6b3acf7ef5e452
I20260810 23:37:40.099627 11663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:40.114494 11663 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:40.115761 11663 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:40.126689 11663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 601f8789a87f4c0c99820bbc2909da2b: Generated new TSK 0
I20260810 23:37:40.127568 11663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:40.134057 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11512
W20260810 23:37:40.135771 11617 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.220:34853
W20260810 23:37:40.136632 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11512: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:37:40.136797 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 11512
W20260810 23:37:40.167991 11486 connection.cc:535] client connection to 127.0.0.1:35529 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35529): Connection reset by peer (error 104)
2026-08-10T23:37:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2274 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:40 dist-test-slave-04bb krb5kdc[11682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:40 dist-test-slave-04bb krb5kdc[11682](info): set up 2 sockets
Aug 10 23:37:40 dist-test-slave-04bb krb5kdc[11682](info): commencing operation
krb5kdc: starting...
W20260810 23:37:41.497565 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.310s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:41 dist-test-slave-04bb krb5kdc[11682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:37:41.825032 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:42.175017 11704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:42.175382 11704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:42.175545 11704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:42.182869 11704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:42.183041 11704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:42.183161 11704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:42.183274 11704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:42.183353 11704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:42.193506 11704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42357
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:42.194885 11704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:42.196281 11704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:42.208716 11711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:42.208640 11712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:42.208721 11717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:42.209583 11704 server_base.cc:1034] running on GCE node
I20260810 23:37:42.210433 11704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:42.212164 11704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:42.213416 11704 hybrid_clock.cc:648] HybridClock initialized: now 1786405062213336 us; error 78 us; skew 500 ppm
I20260810 23:37:42.213789 11704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:42 dist-test-slave-04bb krb5kdc[11682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:42.217221 11704 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:37:42.218717 11704 webserver.cc:460] Webserver started at http://127.0.0.1:38905/ using document root <none> and password file <none>
I20260810 23:37:42.219374 11704 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:42.219475 11704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:42.219723 11704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:42.222762 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "451c56e149024a7898dedcde7e13cbb5"
format_stamp: "Formatted at 2026-08-10 23:37:42 on dist-test-slave-04bb"
server_key: "bc3aac6804a4f8eb500e024e4f039e59"
server_key_iv: "20847f7e75a635ab968a627ed1d1a8c3"
server_key_version: "encryptionkey@0"
I20260810 23:37:42.223496 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "451c56e149024a7898dedcde7e13cbb5"
format_stamp: "Formatted at 2026-08-10 23:37:42 on dist-test-slave-04bb"
server_key: "bc3aac6804a4f8eb500e024e4f039e59"
server_key_iv: "20847f7e75a635ab968a627ed1d1a8c3"
server_key_version: "encryptionkey@0"
I20260810 23:37:42.228964 11704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 23:37:42.232501 11727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:42.233925 11704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:37:42.234125 11704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "451c56e149024a7898dedcde7e13cbb5"
format_stamp: "Formatted at 2026-08-10 23:37:42 on dist-test-slave-04bb"
server_key: "bc3aac6804a4f8eb500e024e4f039e59"
server_key_iv: "20847f7e75a635ab968a627ed1d1a8c3"
server_key_version: "encryptionkey@0"
I20260810 23:37:42.234309 11704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:42.240286 11704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:42.244701 11704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:42.244994 11704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:42.269439 11704 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42357
I20260810 23:37:42.269481 11806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42357 every 8 connection(s)
I20260810 23:37:42.270910 11704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:42.274742 11808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:42.278110 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 11704
I20260810 23:37:42.278314 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:42.278791 9411 external_mini_cluster.cc:1428] Setting key 961086422e8ed2c17a2428646529b473
I20260810 23:37:42.286691 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:42.280647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:54807 (local address 127.0.0.1:42357)
0810 23:37:42.281428 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:42.281440 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:42.281483 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:37:42.283106 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:42.283128 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:42.283485 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:42.283817 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:42.285019 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:42.285026 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:42.285039 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:42.285054 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:42.285119 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:42.285140 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:42.285345 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:42.285527 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 23:37:42.285765 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":342,"threads_started":1}
I20260810 23:37:42.287948 11808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5: Bootstrap starting.
I20260810 23:37:42.290628 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:42.291713 11808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:42.292886 11808 log.cc:824] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:42.296481 11808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5: No bootstrap required, opened a new log
I20260810 23:37:42.297814 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:42.295207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:48143 (local address 127.0.0.1:42357)
0810 23:37:42.295383 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:42.295390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:42.295799 (+ 409us) server_negotiation.cc:407] Connection header received
0810 23:37:42.295994 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:42.295998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:42.296042 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:42.296144 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:42.297135 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:42.297141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:42.297145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:42.297163 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:42.297211 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:42.297225 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:42.297359 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:42.297476 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 23:37:42.297538 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:37:42.305106 11808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451c56e149024a7898dedcde7e13cbb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42357 } }
I20260810 23:37:42.305624 11808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:42.305756 11808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451c56e149024a7898dedcde7e13cbb5, State: Initialized, Role: FOLLOWER
I20260810 23:37:42.306386 11808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451c56e149024a7898dedcde7e13cbb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42357 } }
I20260810 23:37:42.306653 11808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:42.306818 11808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:42.307008 11808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:42.309535 11808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451c56e149024a7898dedcde7e13cbb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42357 } }
I20260810 23:37:42.310079 11808 leader_election.cc:302] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 451c56e149024a7898dedcde7e13cbb5; no voters:
I20260810 23:37:42.310724 11808 leader_election.cc:288] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:42.310820 11823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:42.311722 11823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [term 1 LEADER]: Becoming Leader. State: Replica: 451c56e149024a7898dedcde7e13cbb5, State: Running, Role: LEADER
I20260810 23:37:42.312389 11823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451c56e149024a7898dedcde7e13cbb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42357 } }
I20260810 23:37:42.313288 11808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:42.315455 11826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "451c56e149024a7898dedcde7e13cbb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451c56e149024a7898dedcde7e13cbb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42357 } } }
I20260810 23:37:42.315652 11827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 451c56e149024a7898dedcde7e13cbb5. Latest consensus state: current_term: 1 leader_uuid: "451c56e149024a7898dedcde7e13cbb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451c56e149024a7898dedcde7e13cbb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42357 } } }
I20260810 23:37:42.315908 11826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:42.315927 11827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:42.317818 11838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:42.323210 11838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:42.333020 11838 catalog_manager.cc:1347] Generated new cluster ID: 3b30c6e2c09149339d99a1101cfc02c6
I20260810 23:37:42.333225 11838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:42.358014 11838 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:42.358503 11838 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:42.371282 11838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 451c56e149024a7898dedcde7e13cbb5: Generated new TSK 0
I20260810 23:37:42.372249 11838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:42.386430 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11704
W20260810 23:37:42.387962 11801 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.220:54807
W20260810 23:37:42.388785 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11704: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:37:42.388963 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 11704
2026-08-10T23:37:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2251 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:42 dist-test-slave-04bb krb5kdc[11858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:42 dist-test-slave-04bb krb5kdc[11858](info): set up 2 sockets
Aug 10 23:37:42 dist-test-slave-04bb krb5kdc[11858](info): commencing operation
krb5kdc: starting...
W20260810 23:37:43.821254 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.382s user 0.004s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:44 dist-test-slave-04bb krb5kdc[11858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:37:44.172478 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:39323
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:39323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:44.521272 11874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:44.521550 11874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:44.521760 11874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:44.528962 11874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:44.529091 11874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:44.529203 11874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:44.529313 11874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:44.529395 11874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:44.539186 11874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:39323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:39323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:44.540467 11874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:44.541816 11874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:44.553692 11882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:44.553851 11887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:44.553851 11884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:44.554885 11874 server_base.cc:1034] running on GCE node
I20260810 23:37:44.555697 11874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:44.557130 11874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:44.558362 11874 hybrid_clock.cc:648] HybridClock initialized: now 1786405064558321 us; error 37 us; skew 500 ppm
I20260810 23:37:44.558703 11874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:44 dist-test-slave-04bb krb5kdc[11858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405064, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:44.566960 11874 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:37:44.568703 11874 webserver.cc:460] Webserver started at http://127.9.48.254:42619/ using document root <none> and password file <none>
I20260810 23:37:44.569427 11874 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:44.569572 11874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:44.569906 11874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:44.573043 11874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "6298f493c76c4dcf80e44c2ba5e8f979"
format_stamp: "Formatted at 2026-08-10 23:37:44 on dist-test-slave-04bb"
server_key: "9eb451d6e7674669941758ab86fbae69"
server_key_iv: "d8598cda9b352e73e3c36c71bf525954"
server_key_version: "encryptionkey@0"
I20260810 23:37:44.573825 11874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "6298f493c76c4dcf80e44c2ba5e8f979"
format_stamp: "Formatted at 2026-08-10 23:37:44 on dist-test-slave-04bb"
server_key: "9eb451d6e7674669941758ab86fbae69"
server_key_iv: "d8598cda9b352e73e3c36c71bf525954"
server_key_version: "encryptionkey@0"
I20260810 23:37:44.579049 11874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:37:44.582446 11900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:44.583691 11874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:37:44.583901 11874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "6298f493c76c4dcf80e44c2ba5e8f979"
format_stamp: "Formatted at 2026-08-10 23:37:44 on dist-test-slave-04bb"
server_key: "9eb451d6e7674669941758ab86fbae69"
server_key_iv: "d8598cda9b352e73e3c36c71bf525954"
server_key_version: "encryptionkey@0"
I20260810 23:37:44.584092 11874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:44.599103 11874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:44.602939 11874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:44.603216 11874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:44.623075 11874 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:39323
I20260810 23:37:44.623165 11981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:39323 every 8 connection(s)
I20260810 23:37:44.624473 11874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:44.625890 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 11874
I20260810 23:37:44.626101 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:44.626737 9411 external_mini_cluster.cc:1428] Setting key b49e7bfccd4d6c43be3d7281acd18443
I20260810 23:37:44.629549 11982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:37:44 dist-test-slave-04bb krb5kdc[11858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:37:44.644119 11982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979: Bootstrap starting.
I20260810 23:37:44.647557 11982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:44.649003 11982 log.cc:824] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:44.651353 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:44.628723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50129 (local address 127.9.48.254:39323)
0810 23:37:44.629508 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:44.629520 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:44.629565 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:37:44.631205 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:44.631225 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:44.631591 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:44.631921 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:44.633949 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:44.635407 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:44.637430 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:44.638077 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:44.638219 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:44.643517 (+ 5298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:44.643525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:44.643539 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:44.643574 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:44.646685 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:44.647473 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:44.647480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:44.647486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:44.647649 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:44.648294 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:44.648298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:44.648300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:44.648884 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:44.649189 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:44.649880 (+ 691us) server_negotiation.cc:299] Negotiation successful
0810 23:37:44.650256 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":353,"threads_started":1}
I20260810 23:37:44.652415 11982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979: No bootstrap required, opened a new log
I20260810 23:37:44.655246 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:44.663458 11982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6298f493c76c4dcf80e44c2ba5e8f979" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39323 } }
I20260810 23:37:44.664024 11982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:44.664170 11982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6298f493c76c4dcf80e44c2ba5e8f979, State: Initialized, Role: FOLLOWER
I20260810 23:37:44.664969 11982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6298f493c76c4dcf80e44c2ba5e8f979" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39323 } }
I20260810 23:37:44.665305 11982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:44.665483 11982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:44.665691 11982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:44.668924 11982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6298f493c76c4dcf80e44c2ba5e8f979" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39323 } }
I20260810 23:37:44.669538 11982 leader_election.cc:302] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6298f493c76c4dcf80e44c2ba5e8f979; no voters:
I20260810 23:37:44.670257 11982 leader_election.cc:288] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:44.670373 12004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:44.672369 12004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [term 1 LEADER]: Becoming Leader. State: Replica: 6298f493c76c4dcf80e44c2ba5e8f979, State: Running, Role: LEADER
I20260810 23:37:44.672979 11982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:44.673046 12004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6298f493c76c4dcf80e44c2ba5e8f979" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39323 } }
I20260810 23:37:44.675704 12007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6298f493c76c4dcf80e44c2ba5e8f979. Latest consensus state: current_term: 1 leader_uuid: "6298f493c76c4dcf80e44c2ba5e8f979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6298f493c76c4dcf80e44c2ba5e8f979" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39323 } } }
I20260810 23:37:44.676076 12007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:44.676651 12005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6298f493c76c4dcf80e44c2ba5e8f979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6298f493c76c4dcf80e44c2ba5e8f979" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 39323 } } }
I20260810 23:37:44.676991 12005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:44.680655 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:44.661126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37589 (local address 127.9.48.254:39323)
0810 23:37:44.661354 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:44.661361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:44.661807 (+ 446us) server_negotiation.cc:407] Connection header received
0810 23:37:44.661967 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:44.661973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:44.662056 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:44.662173 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:44.663631 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:44.664991 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:44.666524 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:44.667311 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:44.667465 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:44.670065 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:44.670076 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:44.670085 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:44.670138 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:44.674750 (+ 4612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:44.676183 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:44.676193 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:44.676200 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:44.676467 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:44.677309 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:44.677314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:44.677318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:44.677835 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:44.679531 (+ 1696us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:44.679830 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 23:37:44.680089 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:37:44.680843 12012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:44.688488 12012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:44.697875 12012 catalog_manager.cc:1347] Generated new cluster ID: 9ea3a6c61b45478e85e4b0e49d2b095b
I20260810 23:37:44.697990 12012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:44.721710 12012 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:44.723713 12012 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:44.742745 12012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6298f493c76c4dcf80e44c2ba5e8f979: Generated new TSK 0
I20260810 23:37:44.743899 12012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:44.757731 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 11874
I20260810 23:37:44.759158 11975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:44.852911 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 11874
W20260810 23:37:44.880827 11856 connection.cc:535] client connection to 127.9.48.254:39323 recv error: Network error: failed to read from TLS socket (remote: 127.9.48.254:39323): Connection reset by peer (error 104)
2026-08-10T23:37:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:44 dist-test-slave-04bb krb5kdc[12039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:44 dist-test-slave-04bb krb5kdc[12039](info): set up 2 sockets
Aug 10 23:37:44 dist-test-slave-04bb krb5kdc[12039](info): commencing operation
krb5kdc: starting...
W20260810 23:37:46.216068 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.316s user 0.002s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:46 dist-test-slave-04bb krb5kdc[12039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:37:46.553128 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:44113
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:44113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:46.904093 12058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:46.904395 12058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:46.904613 12058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:46.911877 12058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:46.911976 12058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:46.912089 12058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:46.912209 12058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:46.912292 12058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:46.922382 12058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:44113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:44113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:46.923709 12058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:46.925056 12058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:46.937278 12072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:46.937512 12077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:46.937558 12074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:46.938611 12058 server_base.cc:1034] running on GCE node
I20260810 23:37:46.939468 12058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:46.940951 12058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:46.942152 12058 hybrid_clock.cc:648] HybridClock initialized: now 1786405066942099 us; error 61 us; skew 500 ppm
I20260810 23:37:46.942509 12058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:46 dist-test-slave-04bb krb5kdc[12039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405066, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:46.946115 12058 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:37:46.947712 12058 webserver.cc:460] Webserver started at http://127.9.48.254:42243/ using document root <none> and password file <none>
I20260810 23:37:46.948434 12058 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:46.948596 12058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:46.948922 12058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:46.951939 12058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "dc805cb845684910a89f2f9afdd72d58"
format_stamp: "Formatted at 2026-08-10 23:37:46 on dist-test-slave-04bb"
server_key: "0cdcfce7a4229c591efa9b850d87727c"
server_key_iv: "430d3dfffc26f0509139a19ce4ed6e9c"
server_key_version: "encryptionkey@0"
I20260810 23:37:46.952802 12058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "dc805cb845684910a89f2f9afdd72d58"
format_stamp: "Formatted at 2026-08-10 23:37:46 on dist-test-slave-04bb"
server_key: "0cdcfce7a4229c591efa9b850d87727c"
server_key_iv: "430d3dfffc26f0509139a19ce4ed6e9c"
server_key_version: "encryptionkey@0"
I20260810 23:37:46.958050 12058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:37:46.961453 12088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:46.962780 12058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:37:46.962972 12058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "dc805cb845684910a89f2f9afdd72d58"
format_stamp: "Formatted at 2026-08-10 23:37:46 on dist-test-slave-04bb"
server_key: "0cdcfce7a4229c591efa9b850d87727c"
server_key_iv: "430d3dfffc26f0509139a19ce4ed6e9c"
server_key_version: "encryptionkey@0"
I20260810 23:37:46.963163 12058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:46.993662 12058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:46.998025 12058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:46.998324 12058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:47.018386 12058 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:44113
I20260810 23:37:47.018406 12168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:44113 every 8 connection(s)
I20260810 23:37:47.019791 12058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:47.024122 12169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:47.027441 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 12058
I20260810 23:37:47.027673 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:47.028378 9411 external_mini_cluster.cc:1428] Setting key 26f6d6cd8e08b67334d0b1af27ad5856
I20260810 23:37:47.037940 12169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58: Bootstrap starting.
I20260810 23:37:47.042480 12169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:47.043223 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:47.030680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36817 (local address 127.9.48.254:44113)
0810 23:37:47.031504 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:47.031516 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:47.031560 (+ 44us) server_negotiation.cc:407] Connection header received
0810 23:37:47.033302 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:47.033324 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:47.033734 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:47.034108 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:47.034677 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:47.036031 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:47.037454 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:47.038177 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:47.038367 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:47.039862 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:47.039874 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:47.039896 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:47.040790 (+ 894us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:47.040908 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:47.040953 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:47.041170 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:47.041467 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 23:37:47.041664 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":371,"threads_started":1}
I20260810 23:37:47.044140 12169 log.cc:824] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:47.047699 12169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58: No bootstrap required, opened a new log
I20260810 23:37:47.047850 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:47.055760 12169 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc805cb845684910a89f2f9afdd72d58" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44113 } }
I20260810 23:37:47.056161 12169 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:47.056272 12169 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc805cb845684910a89f2f9afdd72d58, State: Initialized, Role: FOLLOWER
I20260810 23:37:47.056918 12169 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc805cb845684910a89f2f9afdd72d58" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44113 } }
I20260810 23:37:47.057219 12169 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:47.057384 12169 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:47.057583 12169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:47.059883 12169 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc805cb845684910a89f2f9afdd72d58" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44113 } }
I20260810 23:37:47.060105 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:47.052683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57557 (local address 127.9.48.254:44113)
0810 23:37:47.052878 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:47.052888 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:47.053354 (+ 466us) server_negotiation.cc:407] Connection header received
0810 23:37:47.053621 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:47.053629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:47.053693 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:47.053807 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:47.054426 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:47.055281 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:47.056991 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:47.057745 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:47.057933 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:47.059216 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:47.059223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:47.059228 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:47.059259 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:47.059315 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:47.059332 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:47.059464 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:47.059692 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 23:37:47.059748 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:37:47.060580 12169 leader_election.cc:302] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc805cb845684910a89f2f9afdd72d58; no voters:
I20260810 23:37:47.061272 12169 leader_election.cc:288] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:47.061393 12186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:47.062105 12186 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [term 1 LEADER]: Becoming Leader. State: Replica: dc805cb845684910a89f2f9afdd72d58, State: Running, Role: LEADER
I20260810 23:37:47.062784 12186 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc805cb845684910a89f2f9afdd72d58" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44113 } }
I20260810 23:37:47.065112 12186 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc805cb845684910a89f2f9afdd72d58. Latest consensus state: current_term: 1 leader_uuid: "dc805cb845684910a89f2f9afdd72d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc805cb845684910a89f2f9afdd72d58" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44113 } } }
I20260810 23:37:47.065532 12186 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:47.066038 12188 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc805cb845684910a89f2f9afdd72d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc805cb845684910a89f2f9afdd72d58" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44113 } } }
I20260810 23:37:47.066462 12188 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:47.070195 12169 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:47.070431 12194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:47.077129 12194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:47.087930 12194 catalog_manager.cc:1347] Generated new cluster ID: 2931a3299115414e91cce1a893bb6fee
I20260810 23:37:47.088068 12194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:47.113521 12194 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:47.114852 12194 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:47.123799 12194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc805cb845684910a89f2f9afdd72d58: Generated new TSK 0
I20260810 23:37:47.124629 12194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:47.135836 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12058
W20260810 23:37:47.137725 12161 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36817
W20260810 23:37:47.138533 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12058: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:37:47.138710 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 12058
W20260810 23:37:47.166242 12037 connection.cc:535] client connection to 127.9.48.254:44113 recv error: Network error: failed to read from TLS socket (remote: 127.9.48.254:44113): Connection reset by peer (error 104)
2026-08-10T23:37:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:47 dist-test-slave-04bb krb5kdc[12222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:47 dist-test-slave-04bb krb5kdc[12222](info): set up 2 sockets
Aug 10 23:37:47 dist-test-slave-04bb krb5kdc[12222](info): commencing operation
krb5kdc: starting...
W20260810 23:37:48.494704 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.309s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:48 dist-test-slave-04bb krb5kdc[12222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:37:48.840764 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:42561
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:42561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:49.197810 12246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:49.198185 12246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:49.198364 12246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:49.205591 12246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:49.205747 12246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:49.205860 12246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:49.205971 12246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:49.206055 12246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:49.216193 12246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:42561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:42561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:49.217612 12246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:49.218972 12246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:49.231492 12253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:49.231555 12255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:49.232007 12258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:49.232393 12246 server_base.cc:1034] running on GCE node
I20260810 23:37:49.233260 12246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:49.234615 12246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:49.235875 12246 hybrid_clock.cc:648] HybridClock initialized: now 1786405069235809 us; error 52 us; skew 500 ppm
I20260810 23:37:49.236299 12246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:49 dist-test-slave-04bb krb5kdc[12222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405069, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:49.239666 12246 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:37:49.241257 12246 webserver.cc:460] Webserver started at http://127.9.48.254:34045/ using document root <none> and password file <none>
I20260810 23:37:49.241904 12246 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:49.242007 12246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:49.242270 12246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:49.245414 12246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "991892e9760e4797bc72de5f1650ae1c"
format_stamp: "Formatted at 2026-08-10 23:37:49 on dist-test-slave-04bb"
server_key: "d6f4164a454680ac3d8ddf815b430af5"
server_key_iv: "268b074b466b2c8cbe6821196437bb49"
server_key_version: "encryptionkey@0"
I20260810 23:37:49.246171 12246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "991892e9760e4797bc72de5f1650ae1c"
format_stamp: "Formatted at 2026-08-10 23:37:49 on dist-test-slave-04bb"
server_key: "d6f4164a454680ac3d8ddf815b430af5"
server_key_iv: "268b074b466b2c8cbe6821196437bb49"
server_key_version: "encryptionkey@0"
I20260810 23:37:49.251943 12246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 23:37:49.256036 12266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:49.257550 12246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:37:49.257743 12246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "991892e9760e4797bc72de5f1650ae1c"
format_stamp: "Formatted at 2026-08-10 23:37:49 on dist-test-slave-04bb"
server_key: "d6f4164a454680ac3d8ddf815b430af5"
server_key_iv: "268b074b466b2c8cbe6821196437bb49"
server_key_version: "encryptionkey@0"
I20260810 23:37:49.257920 12246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:49.264115 12246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:49.268507 12246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:49.268927 12246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:49.288326 12246 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:42561
I20260810 23:37:49.288368 12360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:42561 every 8 connection(s)
I20260810 23:37:49.289772 12246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:49.293541 12362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:49.294574 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 12246
I20260810 23:37:49.294763 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:49.295374 9411 external_mini_cluster.cc:1428] Setting key fcde3c606f6caa8617a7f5ab716920df
I20260810 23:37:49.303375 12365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:49.297209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39915 (local address 127.9.48.254:42561)
0810 23:37:49.298066 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:49.298077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:49.298122 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:37:49.300011 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:49.300037 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:49.300374 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:49.300766 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:49.301852 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:49.301858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:49.301870 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:49.301886 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:49.301942 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:49.301965 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:49.302133 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:49.302272 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 23:37:49.302474 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":375,"threads_started":1}
I20260810 23:37:49.307004 12362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c: Bootstrap starting.
I20260810 23:37:49.307017 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:49.310222 12362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:49.311475 12362 log.cc:824] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:49.314397 12365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:49.311687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60183 (local address 127.9.48.254:42561)
0810 23:37:49.311869 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:49.311877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:49.312365 (+ 488us) server_negotiation.cc:407] Connection header received
0810 23:37:49.312606 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:49.312610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:49.312686 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:49.312773 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:49.313741 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:49.313748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:49.313752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:49.313771 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:49.313829 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:49.313859 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:49.313977 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:49.314056 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 23:37:49.314117 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 23:37:49.314975 12362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c: No bootstrap required, opened a new log
I20260810 23:37:49.322188 12362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991892e9760e4797bc72de5f1650ae1c" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 42561 } }
I20260810 23:37:49.322546 12362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:49.322664 12362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 991892e9760e4797bc72de5f1650ae1c, State: Initialized, Role: FOLLOWER
I20260810 23:37:49.323237 12362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991892e9760e4797bc72de5f1650ae1c" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 42561 } }
I20260810 23:37:49.323473 12362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:49.323626 12362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:49.323792 12362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:49.326100 12362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991892e9760e4797bc72de5f1650ae1c" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 42561 } }
I20260810 23:37:49.326607 12362 leader_election.cc:302] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 991892e9760e4797bc72de5f1650ae1c; no voters:
I20260810 23:37:49.327183 12362 leader_election.cc:288] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:49.327314 12376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:49.327988 12376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [term 1 LEADER]: Becoming Leader. State: Replica: 991892e9760e4797bc72de5f1650ae1c, State: Running, Role: LEADER
I20260810 23:37:49.328615 12376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991892e9760e4797bc72de5f1650ae1c" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 42561 } }
I20260810 23:37:49.329312 12362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:49.331667 12377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "991892e9760e4797bc72de5f1650ae1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991892e9760e4797bc72de5f1650ae1c" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 42561 } } }
I20260810 23:37:49.331585 12378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 991892e9760e4797bc72de5f1650ae1c. Latest consensus state: current_term: 1 leader_uuid: "991892e9760e4797bc72de5f1650ae1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "991892e9760e4797bc72de5f1650ae1c" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 42561 } } }
I20260810 23:37:49.332124 12377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:49.332159 12378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:49.334115 12390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:49.339047 12390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:49.347718 12390 catalog_manager.cc:1347] Generated new cluster ID: e237f378e5144462a70b1eab76873b78
I20260810 23:37:49.347853 12390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:49.356429 12390 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:49.356921 12390 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:49.369067 12390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 991892e9760e4797bc72de5f1650ae1c: Generated new TSK 0
I20260810 23:37:49.369832 12390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:49.383852 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12246
W20260810 23:37:49.385067 12351 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39915
W20260810 23:37:49.385795 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12246: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:37:49.385957 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 12246
2026-08-10T23:37:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2247 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:49 dist-test-slave-04bb krb5kdc[12412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:49 dist-test-slave-04bb krb5kdc[12412](info): set up 2 sockets
Aug 10 23:37:49 dist-test-slave-04bb krb5kdc[12412](info): commencing operation
krb5kdc: starting...
W20260810 23:37:50.817480 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.385s user 0.007s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:51 dist-test-slave-04bb krb5kdc[12412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:37:51.160226 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:35855
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:44685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:35855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:51.511806 12436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:51.512092 12436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:51.512259 12436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:51.519436 12436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:51.519541 12436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:51.519657 12436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:51.519773 12436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:51.519852 12436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:51.529708 12436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:44685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:35855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:35855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:51.531009 12436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:51.532336 12436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:51.544238 12443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:51.544320 12446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:51.544589 12448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:51.545295 12436 server_base.cc:1034] running on GCE node
I20260810 23:37:51.546033 12436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:51.547405 12436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:51.548628 12436 hybrid_clock.cc:648] HybridClock initialized: now 1786405071548578 us; error 37 us; skew 500 ppm
I20260810 23:37:51.548990 12436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:51 dist-test-slave-04bb krb5kdc[12412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405071, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:51.552218 12436 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:37:51.553890 12436 webserver.cc:460] Webserver started at http://127.9.48.254:42323/ using document root <none> and password file <none>
I20260810 23:37:51.554587 12436 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:51.554747 12436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:51.555068 12436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:51.558172 12436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "6b64a1d8a5e84efbbeb66fee968e0d35"
format_stamp: "Formatted at 2026-08-10 23:37:51 on dist-test-slave-04bb"
server_key: "22de5f14839b0e2dfa88f2aa68e2ce80"
server_key_iv: "c55cf7ca876a28844ff0d2d431f4cb3c"
server_key_version: "encryptionkey@0"
I20260810 23:37:51.558966 12436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "6b64a1d8a5e84efbbeb66fee968e0d35"
format_stamp: "Formatted at 2026-08-10 23:37:51 on dist-test-slave-04bb"
server_key: "22de5f14839b0e2dfa88f2aa68e2ce80"
server_key_iv: "c55cf7ca876a28844ff0d2d431f4cb3c"
server_key_version: "encryptionkey@0"
I20260810 23:37:51.564064 12436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:37:51.567654 12460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:51.569144 12436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:37:51.569343 12436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "6b64a1d8a5e84efbbeb66fee968e0d35"
format_stamp: "Formatted at 2026-08-10 23:37:51 on dist-test-slave-04bb"
server_key: "22de5f14839b0e2dfa88f2aa68e2ce80"
server_key_iv: "c55cf7ca876a28844ff0d2d431f4cb3c"
server_key_version: "encryptionkey@0"
I20260810 23:37:51.569555 12436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:51.585673 12436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:51.589844 12436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:51.590147 12436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:51.609380 12436 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:35855
I20260810 23:37:51.609423 12537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:35855 every 8 connection(s)
I20260810 23:37:51.610761 12436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:51.612720 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 12436
I20260810 23:37:51.612933 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:51.613534 9411 external_mini_cluster.cc:1428] Setting key 08f4753ea9b12407d0a2d88042c8e4aa
I20260810 23:37:51.615252 12542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:37:51 dist-test-slave-04bb krb5kdc[12412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:37:51.629122 12542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35: Bootstrap starting.
I20260810 23:37:51.633293 12542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:51.634630 12542 log.cc:824] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:51.638185 12542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35: No bootstrap required, opened a new log
I20260810 23:37:51.638836 12546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:51.615506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37945 (local address 127.9.48.254:35855)
0810 23:37:51.616311 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:51.616327 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:51.616379 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:37:51.618570 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:51.618596 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:51.619128 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:51.619564 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:51.620997 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:51.623071 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:51.624378 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:51.625435 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:51.625633 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:51.630055 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:51.630066 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:51.630085 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:51.630138 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:51.634224 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:51.634943 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:51.634951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:51.634960 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:51.635176 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:51.635656 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:51.635659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:51.635661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:51.636286 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:51.636594 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:51.637059 (+ 465us) server_negotiation.cc:299] Negotiation successful
0810 23:37:51.637493 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":323,"threads_started":1}
I20260810 23:37:51.642818 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:51.646696 12542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b64a1d8a5e84efbbeb66fee968e0d35" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 35855 } }
I20260810 23:37:51.647071 12542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:51.647192 12542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b64a1d8a5e84efbbeb66fee968e0d35, State: Initialized, Role: FOLLOWER
I20260810 23:37:51.647832 12542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b64a1d8a5e84efbbeb66fee968e0d35" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 35855 } }
I20260810 23:37:51.648090 12542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:51.648252 12542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:51.648450 12542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:51.651484 12542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b64a1d8a5e84efbbeb66fee968e0d35" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 35855 } }
I20260810 23:37:51.652076 12542 leader_election.cc:302] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b64a1d8a5e84efbbeb66fee968e0d35; no voters:
I20260810 23:37:51.652827 12542 leader_election.cc:288] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:51.653035 12556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:51.653808 12556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [term 1 LEADER]: Becoming Leader. State: Replica: 6b64a1d8a5e84efbbeb66fee968e0d35, State: Running, Role: LEADER
I20260810 23:37:51.654452 12556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b64a1d8a5e84efbbeb66fee968e0d35" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 35855 } }
I20260810 23:37:51.656406 12556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b64a1d8a5e84efbbeb66fee968e0d35. Latest consensus state: current_term: 1 leader_uuid: "6b64a1d8a5e84efbbeb66fee968e0d35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b64a1d8a5e84efbbeb66fee968e0d35" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 35855 } } }
I20260810 23:37:51.656738 12556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:51.662880 12558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b64a1d8a5e84efbbeb66fee968e0d35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b64a1d8a5e84efbbeb66fee968e0d35" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 35855 } } }
I20260810 23:37:51.663811 12558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:51.665185 12542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:51.665417 12561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:51.668488 12546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:51.647262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40721 (local address 127.9.48.254:35855)
0810 23:37:51.647460 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:51.647468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:51.648046 (+ 578us) server_negotiation.cc:407] Connection header received
0810 23:37:51.648249 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:51.648254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:51.648326 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:51.648442 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:51.650425 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:51.651176 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:51.653208 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:51.654096 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:51.654254 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:51.658194 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:51.658204 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:51.658212 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:51.658264 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:51.664702 (+ 6438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:51.665637 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:51.665646 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:51.665652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:51.665899 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:51.666419 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:51.666428 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:51.666433 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:51.666939 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:51.667324 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:51.667790 (+ 466us) server_negotiation.cc:299] Negotiation successful
0810 23:37:51.668054 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:37:51.671823 12561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:51.681205 12561 catalog_manager.cc:1347] Generated new cluster ID: f31978782f0d4b91b9f0a34d8d4cc4fe
I20260810 23:37:51.681322 12561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:51.708989 12561 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:51.710886 12561 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:51.718583 12561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b64a1d8a5e84efbbeb66fee968e0d35: Generated new TSK 0
I20260810 23:37:51.719600 12561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:51.734426 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12436
I20260810 23:37:51.735790 12529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:37:51.828320 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 12436
2026-08-10T23:37:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:51 dist-test-slave-04bb krb5kdc[12587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:51 dist-test-slave-04bb krb5kdc[12587](info): set up 2 sockets
Aug 10 23:37:51 dist-test-slave-04bb krb5kdc[12587](info): commencing operation
krb5kdc: starting...
W20260810 23:37:53.248451 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.375s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:53 dist-test-slave-04bb krb5kdc[12587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:37:53.580098 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:45929
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:45929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:53.926627 12605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:53.926976 12605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:53.927165 12605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:53.934579 12605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:53.934751 12605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:53.934870 12605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:53.934985 12605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:53.935065 12605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:53.945282 12605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:45929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:45929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:53.946712 12605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:53.948150 12605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:53.960364 12614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:53.960395 12618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:53.960361 12615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:53.961306 12605 server_base.cc:1034] running on GCE node
I20260810 23:37:53.962113 12605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:53.963510 12605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:53.964745 12605 hybrid_clock.cc:648] HybridClock initialized: now 1786405073964700 us; error 53 us; skew 500 ppm
I20260810 23:37:53.965114 12605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:53 dist-test-slave-04bb krb5kdc[12587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405073, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:53.968385 12605 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:37:53.969936 12605 webserver.cc:460] Webserver started at http://127.9.48.254:37237/ using document root <none> and password file <none>
I20260810 23:37:53.970638 12605 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:53.970793 12605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:53.971122 12605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:53.974269 12605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "7182270c385e4cc695e65ad6e6be8aeb"
format_stamp: "Formatted at 2026-08-10 23:37:53 on dist-test-slave-04bb"
server_key: "39c7a3ff2a541af034d7486b2bd37267"
server_key_iv: "20ebd91145dc7e37723c075c795a345d"
server_key_version: "encryptionkey@0"
I20260810 23:37:53.975055 12605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "7182270c385e4cc695e65ad6e6be8aeb"
format_stamp: "Formatted at 2026-08-10 23:37:53 on dist-test-slave-04bb"
server_key: "39c7a3ff2a541af034d7486b2bd37267"
server_key_iv: "20ebd91145dc7e37723c075c795a345d"
server_key_version: "encryptionkey@0"
I20260810 23:37:53.980254 12605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:37:53.983729 12631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:53.985186 12605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:37:53.985387 12605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "7182270c385e4cc695e65ad6e6be8aeb"
format_stamp: "Formatted at 2026-08-10 23:37:53 on dist-test-slave-04bb"
server_key: "39c7a3ff2a541af034d7486b2bd37267"
server_key_iv: "20ebd91145dc7e37723c075c795a345d"
server_key_version: "encryptionkey@0"
I20260810 23:37:53.985589 12605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:54.000515 12605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:54.005119 12605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:54.005421 12605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:54.024791 12605 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:45929
I20260810 23:37:54.024825 12711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:45929 every 8 connection(s)
I20260810 23:37:54.026244 12605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:54.030165 12712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:54.032491 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 12605
I20260810 23:37:54.032714 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:54.033257 9411 external_mini_cluster.cc:1428] Setting key 13ed89d5007e30da1efd624101f9584d
I20260810 23:37:54.042579 12712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb: Bootstrap starting.
I20260810 23:37:54.045648 12712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:54.047026 12712 log.cc:824] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:54.047591 12719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:54.035154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45513 (local address 127.9.48.254:45929)
0810 23:37:54.036924 (+ 1770us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:54.036937 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:54.036978 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:37:54.038589 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:54.038611 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:54.039001 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:54.039393 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:54.039933 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:54.041393 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:54.042724 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:54.043395 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:54.043593 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:54.044917 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:54.044925 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:54.044937 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:54.045457 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:54.045531 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:54.045559 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:54.045710 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:54.045937 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 23:37:54.046246 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1529,"thread_start_us":1255,"threads_started":1}
I20260810 23:37:54.050338 12712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb: No bootstrap required, opened a new log
I20260810 23:37:54.051680 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:54.059100 12712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7182270c385e4cc695e65ad6e6be8aeb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45929 } }
I20260810 23:37:54.059525 12712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:54.059664 12712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7182270c385e4cc695e65ad6e6be8aeb, State: Initialized, Role: FOLLOWER
I20260810 23:37:54.060317 12712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7182270c385e4cc695e65ad6e6be8aeb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45929 } }
I20260810 23:37:54.060635 12712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:54.060806 12712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:54.061000 12712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:54.062855 12719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:54.056188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58895 (local address 127.9.48.254:45929)
0810 23:37:54.056389 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:54.056397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:54.056919 (+ 522us) server_negotiation.cc:407] Connection header received
0810 23:37:54.057158 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:54.057163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:54.057230 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:54.057365 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:54.057991 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:54.058710 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:54.060220 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:54.060804 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:54.060938 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:54.062090 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:54.062100 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:54.062104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:54.062124 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:54.062162 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:54.062177 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:54.062289 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:54.062451 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 23:37:54.062544 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:37:54.063685 12712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7182270c385e4cc695e65ad6e6be8aeb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45929 } }
I20260810 23:37:54.064256 12712 leader_election.cc:302] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7182270c385e4cc695e65ad6e6be8aeb; no voters:
I20260810 23:37:54.064965 12712 leader_election.cc:288] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:54.065069 12728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:54.065903 12728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [term 1 LEADER]: Becoming Leader. State: Replica: 7182270c385e4cc695e65ad6e6be8aeb, State: Running, Role: LEADER
I20260810 23:37:54.066525 12728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7182270c385e4cc695e65ad6e6be8aeb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45929 } }
I20260810 23:37:54.067276 12712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:54.069499 12729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7182270c385e4cc695e65ad6e6be8aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7182270c385e4cc695e65ad6e6be8aeb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45929 } } }
I20260810 23:37:54.069969 12729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:54.070825 12733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7182270c385e4cc695e65ad6e6be8aeb. Latest consensus state: current_term: 1 leader_uuid: "7182270c385e4cc695e65ad6e6be8aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7182270c385e4cc695e65ad6e6be8aeb" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 45929 } } }
I20260810 23:37:54.071277 12733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:54.075524 12741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:54.081269 12741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:54.091379 12741 catalog_manager.cc:1347] Generated new cluster ID: 55d5460fc12c430faa4785c7790b6148
I20260810 23:37:54.091526 12741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:54.119858 12741 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:54.121249 12741 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:54.128278 12741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7182270c385e4cc695e65ad6e6be8aeb: Generated new TSK 0
I20260810 23:37:54.129134 12741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:54.142112 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12605
W20260810 23:37:54.143726 12703 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45513
W20260810 23:37:54.144485 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12605: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:37:54.144694 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 12605
2026-08-10T23:37:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:54 dist-test-slave-04bb krb5kdc[12761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:54 dist-test-slave-04bb krb5kdc[12761](info): set up 2 sockets
Aug 10 23:37:54 dist-test-slave-04bb krb5kdc[12761](info): commencing operation
krb5kdc: starting...
W20260810 23:37:55.508942 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.320s user 0.004s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:55 dist-test-slave-04bb krb5kdc[12761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:37:55.839830 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:44667
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:44667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:56.193451 12782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:56.193734 12782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:56.193926 12782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:56.201125 12782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:56.201244 12782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:56.201324 12782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:56.201431 12782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:56.201494 12782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:56.211330 12782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:44667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:44667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:56.212731 12782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:56.214058 12782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:56.225930 12788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:56.225845 12795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:56.225895 12791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:56.226759 12782 server_base.cc:1034] running on GCE node
I20260810 23:37:56.227807 12782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:56.229498 12782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:56.230718 12782 hybrid_clock.cc:648] HybridClock initialized: now 1786405076230675 us; error 39 us; skew 500 ppm
I20260810 23:37:56.231088 12782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:56 dist-test-slave-04bb krb5kdc[12761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405076, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:56.238642 12782 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:37:56.240209 12782 webserver.cc:460] Webserver started at http://127.9.48.254:42113/ using document root <none> and password file <none>
I20260810 23:37:56.240983 12782 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:56.241119 12782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:56.241466 12782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:56.244510 12782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "b347da7f76c74162b4e9422da918f298"
format_stamp: "Formatted at 2026-08-10 23:37:56 on dist-test-slave-04bb"
server_key: "d09f92469d9534fa822d5930d3153c9c"
server_key_iv: "5d6d34c9a5075dc453d499f5d183ea44"
server_key_version: "encryptionkey@0"
I20260810 23:37:56.245368 12782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "b347da7f76c74162b4e9422da918f298"
format_stamp: "Formatted at 2026-08-10 23:37:56 on dist-test-slave-04bb"
server_key: "d09f92469d9534fa822d5930d3153c9c"
server_key_iv: "5d6d34c9a5075dc453d499f5d183ea44"
server_key_version: "encryptionkey@0"
I20260810 23:37:56.250625 12782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:37:56.254154 12810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:56.255506 12782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:37:56.255694 12782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "b347da7f76c74162b4e9422da918f298"
format_stamp: "Formatted at 2026-08-10 23:37:56 on dist-test-slave-04bb"
server_key: "d09f92469d9534fa822d5930d3153c9c"
server_key_iv: "5d6d34c9a5075dc453d499f5d183ea44"
server_key_version: "encryptionkey@0"
I20260810 23:37:56.255897 12782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:56.261945 12782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:56.266131 12782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:56.266443 12782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:56.286242 12782 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:44667
I20260810 23:37:56.286295 12894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:44667 every 8 connection(s)
I20260810 23:37:56.287693 12782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:56.291851 12898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:56.293385 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 12782
I20260810 23:37:56.293566 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:56.294021 9411 external_mini_cluster.cc:1428] Setting key fab5b86cb7bf1ed0a807731af93f16b6
I20260810 23:37:56.301589 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:56.295823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54571 (local address 127.9.48.254:44667)
0810 23:37:56.296676 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:56.296687 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:56.296732 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:37:56.298305 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:56.298326 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:56.298664 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:56.298983 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:56.300070 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:56.300077 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:56.300091 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:56.300105 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:56.300165 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:56.300188 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:56.300365 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:56.300458 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 23:37:56.300679 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":375,"threads_started":1}
I20260810 23:37:56.305022 12898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298: Bootstrap starting.
I20260810 23:37:56.305174 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:56.308872 12898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:56.310245 12898 log.cc:824] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:56.313025 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:56.310513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44353 (local address 127.9.48.254:44667)
0810 23:37:56.310725 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:56.310731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:56.311118 (+ 387us) server_negotiation.cc:407] Connection header received
0810 23:37:56.311237 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:56.311244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:56.311318 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:56.311437 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:56.312350 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:56.312355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:56.312358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:37:56.312372 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:56.312409 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:37:56.312423 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:56.312507 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:56.312685 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 23:37:56.312730 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:37:56.313601 12898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298: No bootstrap required, opened a new log
I20260810 23:37:56.320713 12898 raft_consensus.cc:348] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b347da7f76c74162b4e9422da918f298" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44667 } }
I20260810 23:37:56.321086 12898 raft_consensus.cc:374] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:37:56.321190 12898 raft_consensus.cc:729] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b347da7f76c74162b4e9422da918f298, State: Initialized, Role: FOLLOWER
I20260810 23:37:56.321734 12898 consensus_queue.cc:260] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b347da7f76c74162b4e9422da918f298" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44667 } }
I20260810 23:37:56.321979 12898 raft_consensus.cc:388] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:37:56.322124 12898 raft_consensus.cc:482] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:37:56.322290 12898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:56.324404 12898 raft_consensus.cc:504] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b347da7f76c74162b4e9422da918f298" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44667 } }
I20260810 23:37:56.324913 12898 leader_election.cc:302] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b347da7f76c74162b4e9422da918f298; no voters:
I20260810 23:37:56.325598 12898 leader_election.cc:288] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:37:56.325704 12912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:56.326407 12912 raft_consensus.cc:686] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [term 1 LEADER]: Becoming Leader. State: Replica: b347da7f76c74162b4e9422da918f298, State: Running, Role: LEADER
I20260810 23:37:56.327016 12912 consensus_queue.cc:237] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b347da7f76c74162b4e9422da918f298" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44667 } }
I20260810 23:37:56.327988 12898 sys_catalog.cc:564] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:56.330071 12914 sys_catalog.cc:455] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b347da7f76c74162b4e9422da918f298. Latest consensus state: current_term: 1 leader_uuid: "b347da7f76c74162b4e9422da918f298" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b347da7f76c74162b4e9422da918f298" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44667 } } }
I20260810 23:37:56.330480 12914 sys_catalog.cc:458] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:56.331003 12913 sys_catalog.cc:455] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b347da7f76c74162b4e9422da918f298" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b347da7f76c74162b4e9422da918f298" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 44667 } } }
I20260810 23:37:56.331219 12913 sys_catalog.cc:458] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:56.332783 12925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:37:56.338277 12925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:56.346284 12925 catalog_manager.cc:1347] Generated new cluster ID: c84df08faabe4294abbdbb98f009b4b9
I20260810 23:37:56.346407 12925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:56.364509 12925 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:56.365033 12925 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:56.374092 12925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b347da7f76c74162b4e9422da918f298: Generated new TSK 0
I20260810 23:37:56.375044 12925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:56.389751 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12782
W20260810 23:37:56.391175 12886 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54571
W20260810 23:37:56.391927 9411 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12782: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 23:37:56.392078 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 12782
2026-08-10T23:37:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2252 ms)
[----------] 18 tests from AuthTokenIssuingTest (41998 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:37:56 dist-test-slave-04bb krb5kdc[12953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:37:56 dist-test-slave-04bb krb5kdc[12953](info): set up 2 sockets
Aug 10 23:37:56 dist-test-slave-04bb krb5kdc[12953](info): commencing operation
krb5kdc: starting...
W20260810 23:37:57.749850 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.307s user 0.004s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:37:57 dist-test-slave-04bb krb5kdc[12953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:37:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:37:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:37:58.084017 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:41411
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:40201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:41411,127.9.48.253:43953,127.9.48.252:43229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:37:58.434584 12979 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:37:58.434898 12979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:58.434964 12979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:58.435048 12979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:58.442077 12979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:58.442157 12979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:58.442214 12979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:58.442265 12979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:58.442319 12979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:58.451879 12979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:40201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:41411,127.9.48.253:43953,127.9.48.252:43229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:41411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:58.453079 12979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:58.454331 12979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:58.466006 12988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:58.466125 12993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:58.466325 12991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:58.467410 12979 server_base.cc:1034] running on GCE node
I20260810 23:37:58.468194 12979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:58.469537 12979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:58.470783 12979 hybrid_clock.cc:648] HybridClock initialized: now 1786405078470709 us; error 40 us; skew 500 ppm
I20260810 23:37:58.471076 12979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:58 dist-test-slave-04bb krb5kdc[12953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405078, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:58.474255 12979 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:37:58.475708 12979 webserver.cc:460] Webserver started at http://127.9.48.254:35257/ using document root <none> and password file <none>
I20260810 23:37:58.476405 12979 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:58.476521 12979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:58.476886 12979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:58.479843 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "7b40b77f158947a497e8a65a3e0d3378"
format_stamp: "Formatted at 2026-08-10 23:37:58 on dist-test-slave-04bb"
server_key: "2806775d7de4770b866b39dddf326732"
server_key_iv: "038877c37ca0ee715ef0ee3f4ee60f73"
server_key_version: "encryptionkey@0"
I20260810 23:37:58.480688 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "7b40b77f158947a497e8a65a3e0d3378"
format_stamp: "Formatted at 2026-08-10 23:37:58 on dist-test-slave-04bb"
server_key: "2806775d7de4770b866b39dddf326732"
server_key_iv: "038877c37ca0ee715ef0ee3f4ee60f73"
server_key_version: "encryptionkey@0"
I20260810 23:37:58.485746 12979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 23:37:58.489195 13005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:58.490545 12979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:37:58.490728 12979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "7b40b77f158947a497e8a65a3e0d3378"
format_stamp: "Formatted at 2026-08-10 23:37:58 on dist-test-slave-04bb"
server_key: "2806775d7de4770b866b39dddf326732"
server_key_iv: "038877c37ca0ee715ef0ee3f4ee60f73"
server_key_version: "encryptionkey@0"
I20260810 23:37:58.490963 12979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:58.531457 12979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:58.535554 12979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:58.535844 12979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:58.554221 12979 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:41411
I20260810 23:37:58.554245 13084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:41411 every 8 connection(s)
I20260810 23:37:58.555608 12979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:37:58.559511 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:58.559945 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 12979
I20260810 23:37:58.560186 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:37:58.561065 9411 external_mini_cluster.cc:1428] Setting key 022c5d7757ce5d21ac4113f7f5184d18
I20260810 23:37:58.569866 13085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:37:58 dist-test-slave-04bb krb5kdc[12953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405078, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.48.253@KRBTEST.COM; defaulting to no policy
I20260810 23:37:58.600293 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:58.574023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:41411 (local address 127.0.0.1:52316)
0810 23:37:58.578687 (+ 4664us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:58.578730 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:58.581572 (+ 2842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:58.582547 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:58.582556 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:58.583397 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:58.583963 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:58.583991 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:58.585610 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:58.585618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:58.587579 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:58.587597 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:58.588650 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:58.588660 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:58.588898 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:58.589621 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:37:58.589655 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:58.592437 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:58.596107 (+ 3670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:58.596118 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:58.596130 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:58.596569 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:58.596974 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:58.596982 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:58.596986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:58.597145 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:58.597880 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:58.597893 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:58.598126 (+ 233us) client_negotiation.cc:769] Sending connection context
0810 23:37:58.598369 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 23:37:58.598963 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4405,"mutex_wait_us":162,"thread_start_us":4150,"threads_started":1}
I20260810 23:37:58.600311 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:58.574023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.9.48.254:41411)
0810 23:37:58.578292 (+ 4269us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:58.578309 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:58.578882 (+ 573us) server_negotiation.cc:407] Connection header received
0810 23:37:58.582048 (+ 3166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:58.582053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:58.582289 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:58.582451 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:58.584188 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:58.585443 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:58.587797 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:58.588375 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:58.588508 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:58.592675 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:58.592683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:58.592696 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:58.592724 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:58.595982 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:58.596697 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:58.596705 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:58.596709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:58.596867 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:58.597257 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:58.597262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:58.597266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:58.597738 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:58.597960 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:58.598504 (+ 544us) server_negotiation.cc:299] Negotiation successful
0810 23:37:58.598844 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":3986,"thread_start_us":3739,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.253@KRBTEST.COM" created.
I20260810 23:37:58.603389 13085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:37:58.604558 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:58.604123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:43953 (local address 127.0.0.1:40746)
0810 23:37:58.604306 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:58.604412 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 23:37:58.604847 13007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
I20260810 23:37:58.606401 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:58.606039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:43953 (local address 127.0.0.1:40756)
0810 23:37:58.606180 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:58.606249 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 23:37:58.606594 13085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.253:43953: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.253.keytab.
Entry for principal kudu/127.9.48.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.253.keytab.
I20260810 23:37:58.639706 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.253:43953
--webserver_interface=127.9.48.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.253.keytab
--principal=kudu/127.9.48.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:40201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:41411,127.9.48.253:43953,127.9.48.252:43229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:37:58.655946 13085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } attempt: 1
I20260810 23:37:58.657418 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:58.656792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:43953 (local address 127.0.0.1:40768)
0810 23:37:58.657057 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:58.657219 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 23:37:58.658607 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:58.658122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:43953 (local address 127.0.0.1:40782)
0810 23:37:58.658306 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:58.658397 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:37:58.658859 13085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.253:43953: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
I20260810 23:37:58.727152 13085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } attempt: 2
I20260810 23:37:58.728616 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:58.727941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:43953 (local address 127.0.0.1:40784)
0810 23:37:58.728219 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:58.728357 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 23:37:58.729923 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:58.729401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:43953 (local address 127.0.0.1:40796)
0810 23:37:58.729585 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:58.729716 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 23:37:58.730149 13085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.253:43953: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
I20260810 23:37:58.821450 13085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } attempt: 3
I20260810 23:37:58.822780 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:58.822220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:43953 (local address 127.0.0.1:40798)
0810 23:37:58.822470 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:58.822586 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260810 23:37:58.823835 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:58.823417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:43953 (local address 127.0.0.1:40800)
0810 23:37:58.823569 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:58.823663 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:37:58.824019 13085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.253:43953: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
I20260810 23:37:58.967307 13085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } attempt: 4
I20260810 23:37:58.969007 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:58.968218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:43953 (local address 127.0.0.1:40808)
0810 23:37:58.968650 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:58.968793 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
I20260810 23:37:58.970193 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:58.969739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:43953 (local address 127.0.0.1:40812)
0810 23:37:58.969910 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:58.970020 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 23:37:58.970443 13085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.253:43953: Network error: Client connection negotiation failed: client connection to 127.9.48.253:43953: connect: Connection refused (error 111)
W20260810 23:37:59.000031 13097 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:37:59.000360 13097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:59.000434 13097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:59.000515 13097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:59.007716 13097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:59.007799 13097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:59.007872 13097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:59.007925 13097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:59.007993 13097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:59.017742 13097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:40201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:41411,127.9.48.253:43953,127.9.48.252:43229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.253.keytab
--principal=kudu/127.9.48.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.253:43953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.48.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:59.018944 13097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:59.020213 13097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:59.032287 13108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:59.032259 13110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:59.032418 13106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:59.033335 13097 server_base.cc:1034] running on GCE node
I20260810 23:37:59.034126 13097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:59.035475 13097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:59.036645 13097 hybrid_clock.cc:648] HybridClock initialized: now 1786405079036555 us; error 57 us; skew 500 ppm
I20260810 23:37:59.036939 13097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:59 dist-test-slave-04bb krb5kdc[12953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405079, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:59.040246 13097 init.cc:377] Logged in from keytab as kudu/127.9.48.253@KRBTEST.COM (short username kudu)
I20260810 23:37:59.041818 13097 webserver.cc:460] Webserver started at http://127.9.48.253:42133/ using document root <none> and password file <none>
I20260810 23:37:59.042550 13097 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:59.042639 13097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:59.042992 13097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:59.046056 13097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/data/instance:
uuid: "46a12ee77ff94e2da8d9e95a44d13f9d"
format_stamp: "Formatted at 2026-08-10 23:37:59 on dist-test-slave-04bb"
server_key: "fee69cd7da1299ff1db6bd3358e8a53b"
server_key_iv: "728fdf14349871a4fec37b173138b71a"
server_key_version: "encryptionkey@0"
I20260810 23:37:59.046869 13097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/wal/instance:
uuid: "46a12ee77ff94e2da8d9e95a44d13f9d"
format_stamp: "Formatted at 2026-08-10 23:37:59 on dist-test-slave-04bb"
server_key: "fee69cd7da1299ff1db6bd3358e8a53b"
server_key_iv: "728fdf14349871a4fec37b173138b71a"
server_key_version: "encryptionkey@0"
I20260810 23:37:59.052501 13097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:37:59.056401 13127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:59.057891 13097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 23:37:59.058089 13097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/wal
uuid: "46a12ee77ff94e2da8d9e95a44d13f9d"
format_stamp: "Formatted at 2026-08-10 23:37:59 on dist-test-slave-04bb"
server_key: "fee69cd7da1299ff1db6bd3358e8a53b"
server_key_iv: "728fdf14349871a4fec37b173138b71a"
server_key_version: "encryptionkey@0"
I20260810 23:37:59.058349 13097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:59.077318 13097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:59.081166 13097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:59.081440 13097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:59.100351 13097 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.253:43953
I20260810 23:37:59.100375 13211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.253:43953 every 8 connection(s)
I20260810 23:37:59.101861 13097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/data/info.pb
I20260810 23:37:59.103564 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 13097
I20260810 23:37:59.103791 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-1/wal/instance
I20260810 23:37:59.104239 9411 external_mini_cluster.cc:1428] Setting key d4ccb6fdf038b3d5379c971972c28f11
I20260810 23:37:59.105820 13212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:59.112700 13212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:37:59 dist-test-slave-04bb krb5kdc[12953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405079, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.253@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:37:59.135071 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.115254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.9.48.254:41411)
0810 23:37:59.115793 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:59.115800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:59.116800 (+ 1000us) server_negotiation.cc:407] Connection header received
0810 23:37:59.118666 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:59.118675 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:59.118773 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:59.118900 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:59.121107 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.121866 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.124321 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.124779 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.124876 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.129202 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:59.129207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:59.129210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:59.129254 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:59.131975 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.132767 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.132772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.132774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.132893 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.133328 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.133332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.133334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.133601 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:59.133775 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:59.134463 (+ 688us) server_negotiation.cc:299] Negotiation successful
0810 23:37:59.134688 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":263,"threads_started":1}
I20260810 23:37:59.136027 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.115748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:41411 (local address 127.0.0.1:52320)
0810 23:37:59.116611 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.116672 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:59.118375 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:59.119231 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:59.119244 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:59.120022 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:59.120878 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.120907 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.122036 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.122044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.124160 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.124178 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.125006 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.125015 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.125357 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.126144 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:37:59.126191 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:59.128985 (+ 2794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:59.132126 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.132135 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.132150 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.132640 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.133005 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.133010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.133014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.133193 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.133734 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:59.133743 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:59.134095 (+ 352us) client_negotiation.cc:769] Sending connection context
0810 23:37:59.134355 (+ 260us) client_negotiation.cc:240] Negotiation successful
0810 23:37:59.134880 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":367,"threads_started":1}
I20260810 23:37:59.138041 13212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.48.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.252@KRBTEST.COM" created.
Aug 10 23:37:59 dist-test-slave-04bb krb5kdc[12953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405079, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.253@KRBTEST.COM for kudu/127.9.48.253@KRBTEST.COM
I20260810 23:37:59.155249 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.138883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:43953 (local address 127.0.0.1:40828)
0810 23:37:59.139043 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.139069 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:59.139171 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:59.140065 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:59.140070 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:59.140494 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:59.140695 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.140704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.142288 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.142293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.143117 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.143128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.143948 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.143954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.144128 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.147314 (+ 3186us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:37:59.147340 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:59.149506 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:59.152763 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.152771 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.152775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.153156 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.153524 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.153529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.153533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.153672 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.154258 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:59.154265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:59.154418 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 23:37:59.154567 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 23:37:59.154834 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 23:37:59.155306 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.139021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.9.48.253:43953)
0810 23:37:59.139575 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:59.139581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:59.139624 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:37:59.139697 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:59.139707 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:59.139918 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:59.140086 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:59.140873 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.142142 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.143257 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.143782 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.143912 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.149718 (+ 5806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:59.149730 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:59.149743 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:59.149771 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:59.152642 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.153286 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.153299 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.153302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.153411 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.153790 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.153794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.153798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.154137 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:59.154357 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:59.154706 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 23:37:59.154927 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":288,"threads_started":1}
I20260810 23:37:59.157783 13212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:37:59.159005 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.158546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60156)
0810 23:37:59.158737 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.158863 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 23:37:59.159272 13131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
I20260810 23:37:59.160933 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.160609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60170)
0810 23:37:59.160743 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.160830 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:37:59.161164 13212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:43229: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.252.keytab.
Entry for principal kudu/127.9.48.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.252.keytab.
I20260810 23:37:59.183235 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.252:43229
--webserver_interface=127.9.48.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.252.keytab
--principal=kudu/127.9.48.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:40201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:41411,127.9.48.253:43953,127.9.48.252:43229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:37:59.210470 13212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } attempt: 1
I20260810 23:37:59.211860 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.211200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60174)
0810 23:37:59.211476 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.211623 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 23:37:59.213125 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.212629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60190)
0810 23:37:59.212847 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.212960 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:37:59.213380 13212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:43229: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
I20260810 23:37:59.269716 13085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } attempt: 5
Aug 10 23:37:59 dist-test-slave-04bb krb5kdc[12953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405078, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for kudu/127.9.48.253@KRBTEST.COM
I20260810 23:37:59.281618 13212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } attempt: 2
I20260810 23:37:59.282780 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.282258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60196)
0810 23:37:59.282501 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.282575 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260810 23:37:59.283289 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.270467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:43953 (local address 127.0.0.1:40842)
0810 23:37:59.270672 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.270699 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:59.270838 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:59.271214 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:59.271222 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:59.271638 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:59.271830 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.271839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.272888 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.272894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.273765 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.273776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.274526 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.274532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.274745 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.275390 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:37:59.275408 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:59.277496 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:59.280596 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.280601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.280605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.280995 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.281382 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.281387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.281391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.281549 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.282249 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:59.282255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:59.282447 (+ 192us) client_negotiation.cc:769] Sending connection context
0810 23:37:59.282652 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 23:37:59.282956 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 23:37:59.283329 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.270591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40842 (local address 127.9.48.253:43953)
0810 23:37:59.270819 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:59.270825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:59.270848 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:37:59.270963 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:59.270968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:59.271059 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:59.271167 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:59.271980 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.272755 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.273920 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.274375 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.274506 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.277692 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:59.277697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:59.277701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:59.277733 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:59.280417 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.281111 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.281119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.281122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.281266 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.281659 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.281665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.281668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.282109 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:59.282276 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:59.282786 (+ 510us) server_negotiation.cc:299] Negotiation successful
0810 23:37:59.283033 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 23:37:59.284466 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.283954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60204)
0810 23:37:59.284171 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.284278 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 23:37:59.284837 13212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:43229: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
I20260810 23:37:59.284871 13085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:37:59.286106 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.285643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60220)
0810 23:37:59.285891 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.285967 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260810 23:37:59.287165 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.286831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60226)
0810 23:37:59.286984 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.287055 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 23:37:59.287434 13085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:43229: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
I20260810 23:37:59.338668 13085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } attempt: 1
I20260810 23:37:59.339908 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.339430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60236)
0810 23:37:59.339638 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.339731 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 23:37:59.340960 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.340623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60244)
0810 23:37:59.340780 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.340851 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 23:37:59.341172 13085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:43229: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
I20260810 23:37:59.376099 13212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } attempt: 3
I20260810 23:37:59.377404 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.376832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60258)
0810 23:37:59.377083 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.377203 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 23:37:59.378527 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.378058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60270)
0810 23:37:59.378291 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.378385 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260810 23:37:59.378785 13212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:43229: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
I20260810 23:37:59.409389 13085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } attempt: 2
I20260810 23:37:59.410609 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.410096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60274)
0810 23:37:59.410332 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.410432 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 23:37:59.411631 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.411285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60288)
0810 23:37:59.411448 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.411521 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 23:37:59.411864 13085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:43229: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
I20260810 23:37:59.518069 13085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } attempt: 3
I20260810 23:37:59.519261 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.518765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60290)
0810 23:37:59.518988 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.519083 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 23:37:59.520270 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.519882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60296)
0810 23:37:59.520100 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.520164 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260810 23:37:59.520452 13085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:43229: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
I20260810 23:37:59.522010 13212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } attempt: 4
I20260810 23:37:59.523098 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.522606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60310)
0810 23:37:59.522883 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.522963 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260810 23:37:59.524091 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.523766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60320)
0810 23:37:59.523923 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.523991 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:37:59.524274 13212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:43229: Network error: Client connection negotiation failed: client connection to 127.9.48.252:43229: connect: Connection refused (error 111)
W20260810 23:37:59.538340 13224 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:37:59.538700 13224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:37:59.538821 13224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:37:59.538950 13224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:37:59.546526 13224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:37:59.546643 13224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:37:59.546772 13224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:37:59.546857 13224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:37:59.546964 13224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:37:59.557181 13224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:40201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:41411,127.9.48.253:43953,127.9.48.252:43229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.252.keytab
--principal=kudu/127.9.48.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.252:43229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.48.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:37:59.558503 13224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:37:59.559847 13224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:37:59.572032 13242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:59.572098 13245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:37:59.571993 13240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:37:59.572996 13224 server_base.cc:1034] running on GCE node
I20260810 23:37:59.573767 13224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:37:59.575088 13224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:37:59.576339 13224 hybrid_clock.cc:648] HybridClock initialized: now 1786405079576271 us; error 43 us; skew 500 ppm
I20260810 23:37:59.576718 13224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:37:59 dist-test-slave-04bb krb5kdc[12953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405079, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:37:59.579777 13224 init.cc:377] Logged in from keytab as kudu/127.9.48.252@KRBTEST.COM (short username kudu)
I20260810 23:37:59.581310 13224 webserver.cc:460] Webserver started at http://127.9.48.252:41441/ using document root <none> and password file <none>
I20260810 23:37:59.581991 13224 fs_manager.cc:359] Metadata directory not provided
I20260810 23:37:59.582121 13224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:37:59.582451 13224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:37:59.585521 13224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/data/instance:
uuid: "b7c44e77e706448881ba441bb244aa5d"
format_stamp: "Formatted at 2026-08-10 23:37:59 on dist-test-slave-04bb"
server_key: "eb67b5b7676407d38d899ee6aafaad3e"
server_key_iv: "efc1caf99e0b89624aef982d9f32e35d"
server_key_version: "encryptionkey@0"
I20260810 23:37:59.586377 13224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/wal/instance:
uuid: "b7c44e77e706448881ba441bb244aa5d"
format_stamp: "Formatted at 2026-08-10 23:37:59 on dist-test-slave-04bb"
server_key: "eb67b5b7676407d38d899ee6aafaad3e"
server_key_iv: "efc1caf99e0b89624aef982d9f32e35d"
server_key_version: "encryptionkey@0"
I20260810 23:37:59.591490 13224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 23:37:59.594911 13257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:37:59.596244 13224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:37:59.596428 13224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/wal
uuid: "b7c44e77e706448881ba441bb244aa5d"
format_stamp: "Formatted at 2026-08-10 23:37:59 on dist-test-slave-04bb"
server_key: "eb67b5b7676407d38d899ee6aafaad3e"
server_key_iv: "efc1caf99e0b89624aef982d9f32e35d"
server_key_version: "encryptionkey@0"
I20260810 23:37:59.596706 13224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:37:59.620342 13224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:37:59.624176 13224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:37:59.624496 13224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:37:59.643055 13224 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.252:43229
I20260810 23:37:59.643076 13341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.252:43229 every 8 connection(s)
I20260810 23:37:59.644362 13224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/data/info.pb
I20260810 23:37:59.647277 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 13224
I20260810 23:37:59.647468 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786404698557218-9411-0/minicluster-data/master-2/wal/instance
I20260810 23:37:59.647959 9411 external_mini_cluster.cc:1428] Setting key c14d9f9d4d4e2df9a7a3b4cc80d08714
I20260810 23:37:59.648087 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:37:59.658222 13343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:37:59 dist-test-slave-04bb krb5kdc[12953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.252@KRBTEST.COM
Aug 10 23:37:59 dist-test-slave-04bb krb5kdc[12953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405079, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.252@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:37:59.673043 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.649847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.9.48.252:43229)
0810 23:37:59.650652 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:59.650667 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:59.650725 (+ 58us) server_negotiation.cc:407] Connection header received
0810 23:37:59.653045 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:59.653076 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:59.653585 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:59.654052 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:59.655547 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.657546 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.658851 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.659844 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.660019 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.665089 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:59.665100 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:59.665111 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:59.665146 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:59.668646 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.669498 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.669510 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.669522 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.669716 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.670118 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.670123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.670126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.670780 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:59.670993 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:59.671491 (+ 498us) server_negotiation.cc:299] Negotiation successful
0810 23:37:59.671922 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":322,"threads_started":1}
I20260810 23:37:59.676932 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.660951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:41411 (local address 127.0.0.1:52332)
0810 23:37:59.661702 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.661751 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:59.661894 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:59.662667 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:59.662674 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:59.663394 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:59.663727 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.663743 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.664877 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.664886 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.666011 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.666024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.667110 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.667116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.667255 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.667842 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:37:59.667865 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:59.670541 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:59.674098 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.674106 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.674111 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.674657 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.675049 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.675054 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.675057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.675236 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.675839 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:59.675846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:59.676029 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 23:37:59.676223 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 23:37:59.676593 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":382,"threads_started":1}
I20260810 23:37:59.677170 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.660981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52332 (local address 127.9.48.254:41411)
0810 23:37:59.662339 (+ 1358us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:59.662343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:59.662359 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:37:59.662408 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:59.662412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:59.662483 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:59.662602 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:59.663902 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.664718 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.666194 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.666906 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.667044 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.670727 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:59.670732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:59.670735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:59.670769 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:59.673974 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.674790 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.674795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.674797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.674939 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.675346 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.675349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.675351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.675702 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:59.675853 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:59.676377 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 23:37:59.676609 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1248,"thread_start_us":1488,"threads_started":1}
I20260810 23:37:59.678479 13343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:37:59 dist-test-slave-04bb krb5kdc[12953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405079, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.252@KRBTEST.COM for kudu/127.9.48.253@KRBTEST.COM
Aug 10 23:37:59 dist-test-slave-04bb krb5kdc[12953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:37:59.693348 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.679106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:43953 (local address 127.0.0.1:40858)
0810 23:37:59.679460 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.679473 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:59.679591 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:59.680074 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:59.680079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:59.680555 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:59.680728 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.680739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.682459 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.682467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.683603 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.683620 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.684920 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.684924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.685044 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.685545 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:37:59.685561 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:59.687626 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:59.690794 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.690802 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.690808 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.691299 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.691651 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.691657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.691662 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.691843 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.692431 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:59.692434 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:59.692616 (+ 182us) client_negotiation.cc:769] Sending connection context
0810 23:37:59.692774 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 23:37:59.693030 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277}
I20260810 23:37:59.693485 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.679190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.9.48.253:43953)
0810 23:37:59.679364 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:59.679368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:59.679545 (+ 177us) server_negotiation.cc:407] Connection header received
0810 23:37:59.679700 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:59.679705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:59.679779 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:59.679883 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:59.680915 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.682289 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.683821 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.684754 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.684878 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.687849 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:59.687856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:59.687862 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:59.687905 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:59.690661 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.691409 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.691413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.691415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.691542 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.691954 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.691958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.691961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.692293 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:59.692459 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:59.692910 (+ 451us) server_negotiation.cc:299] Negotiation successful
0810 23:37:59.693158 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:37:59.694371 13343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:37:59.695107 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.676518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.9.48.254:41411)
0810 23:37:59.677476 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:59.677489 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:59.677513 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:37:59.677588 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:59.677595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:59.677685 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:59.677803 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:59.679894 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.681013 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.682190 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.682747 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.682881 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.688794 (+ 5913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:59.688802 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:59.688807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:59.688843 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:59.691833 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.692617 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.692622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.692625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.692795 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.693222 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.693228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.693232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.693871 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:59.694141 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:59.694465 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 23:37:59.694714 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":8128,"threads_started":1}
I20260810 23:37:59.697678 13085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } attempt: 4
Aug 10 23:37:59 dist-test-slave-04bb krb5kdc[12953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405079, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.252@KRBTEST.COM for kudu/127.9.48.252@KRBTEST.COM
Aug 10 23:37:59 dist-test-slave-04bb krb5kdc[12953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405078, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for kudu/127.9.48.252@KRBTEST.COM
Aug 10 23:37:59 dist-test-slave-04bb krb5kdc[12953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.253@KRBTEST.COM
I20260810 23:37:59.713191 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.695162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60344)
0810 23:37:59.695552 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.695566 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:59.695733 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:59.696018 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:59.696021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:59.696469 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:59.696683 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.696699 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.698377 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.698381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.699735 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.699746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.700915 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.700922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.701152 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.703186 (+ 2034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:37:59.703204 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:59.706122 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:59.710380 (+ 4258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.710385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.710392 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.710957 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.711385 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.711390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.711393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.711558 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.712152 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:59.712157 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:59.712296 (+ 139us) client_negotiation.cc:769] Sending connection context
0810 23:37:59.712448 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 23:37:59.712779 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313}
I20260810 23:37:59.713263 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.695162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.9.48.252:43229)
0810 23:37:59.695328 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:59.695335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:59.695630 (+ 295us) server_negotiation.cc:407] Connection header received
0810 23:37:59.695845 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:59.695850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:59.695911 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:59.696001 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:59.696851 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.697905 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.699916 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.700729 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.700861 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.706368 (+ 5507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:59.706376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:59.706380 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:59.706421 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:59.710213 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.711083 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.711089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.711093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.711244 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.711686 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.711692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.711694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.712036 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:59.712195 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:59.712544 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 23:37:59.712940 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":166,"server-negotiator.queue_time_us":64}
I20260810 23:37:59.714226 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.698214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60354)
0810 23:37:59.698664 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.698682 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:59.698782 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:59.699357 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:59.699362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:59.699832 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:59.700018 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.700030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.701671 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.701677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.702898 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.702915 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.703818 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.703824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.704028 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.704607 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:37:59.704625 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:59.706842 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:59.710961 (+ 4119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.710967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.710970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.711408 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.711841 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.711847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.711851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.711997 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.712722 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:59.712726 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:59.712920 (+ 194us) client_negotiation.cc:769] Sending connection context
0810 23:37:59.713067 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 23:37:59.713862 (+ 795us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319}
I20260810 23:37:59.715214 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.696932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40870 (local address 127.9.48.253:43953)
0810 23:37:59.697128 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:59.697135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:59.697234 (+ 99us) server_negotiation.cc:407] Connection header received
0810 23:37:59.697385 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:59.697389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:59.697441 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:59.697520 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:59.699149 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.700232 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.702025 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.702435 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.702535 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.708227 (+ 5692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:59.708235 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:59.708241 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:59.708287 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:59.711704 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.712876 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.712882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.712885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.713051 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.713537 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.713541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.713543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.713870 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:59.714155 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:59.714701 (+ 546us) server_negotiation.cc:299] Negotiation successful
0810 23:37:59.714948 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:37:59.715330 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.698316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60354 (local address 127.9.48.252:43229)
0810 23:37:59.698977 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:59.698982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:59.699005 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:37:59.699048 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:59.699053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:59.699125 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:59.699288 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:59.700168 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.701502 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.703072 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.703631 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.703797 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.707053 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:59.707058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:59.707061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:59.707088 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:59.710802 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.711566 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.711570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.711573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.711721 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.712106 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.712112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.712115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.712453 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:59.714568 (+ 2115us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:59.714706 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 23:37:59.714909 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":518,"threads_started":1}
I20260810 23:37:59.720110 13085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378: Bootstrap starting.
I20260810 23:37:59.720157 13343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d: Bootstrap starting.
I20260810 23:37:59.720661 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:37:59.723409 13085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:59.723942 13343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:59.724735 13085 log.cc:824] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:59.725252 13343 log.cc:824] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:37:59.728031 13343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d: No bootstrap required, opened a new log
I20260810 23:37:59.728438 13085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378: No bootstrap required, opened a new log
I20260810 23:37:59.724331 13041 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7b40b77f158947a497e8a65a3e0d3378"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52340
I20260810 23:37:59.735087 13343 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } }
I20260810 23:37:59.735455 13343 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7c44e77e706448881ba441bb244aa5d, State: Initialized, Role: FOLLOWER
I20260810 23:37:59.735760 13085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } }
I20260810 23:37:59.736061 13085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b40b77f158947a497e8a65a3e0d3378, State: Initialized, Role: FOLLOWER
I20260810 23:37:59.735988 13343 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } }
I20260810 23:37:59.736613 13085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } }
I20260810 23:37:59.737018 13359 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } } }
I20260810 23:37:59.737565 13359 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:37:59.737936 13360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } } }
I20260810 23:37:59.738408 13360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:37:59.738430 13343 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:59.738780 13085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:59.738895 13041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 23:37:59.739087 13041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:59.743309 13041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } }
I20260810 23:37:59.747303 13304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7b40b77f158947a497e8a65a3e0d3378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b7c44e77e706448881ba441bb244aa5d"
I20260810 23:37:59.747859 13304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:59.752313 13304 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b40b77f158947a497e8a65a3e0d3378 in term 1.
I20260810 23:37:59.752662 13041 leader_election.cc:288] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [CANDIDATE]: Term 1 election: Requested vote from peers 46a12ee77ff94e2da8d9e95a44d13f9d (127.9.48.253:43953), b7c44e77e706448881ba441bb244aa5d (127.9.48.252:43229)
I20260810 23:37:59.753227 13010 leader_election.cc:302] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b40b77f158947a497e8a65a3e0d3378, b7c44e77e706448881ba441bb244aa5d; no voters:
I20260810 23:37:59.753762 13360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:37:59.757396 13360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [term 1 LEADER]: Becoming Leader. State: Replica: 7b40b77f158947a497e8a65a3e0d3378, State: Running, Role: LEADER
I20260810 23:37:59.758209 13360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } }
I20260810 23:37:59.777266 13382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b40b77f158947a497e8a65a3e0d3378. Latest consensus state: current_term: 1 leader_uuid: "7b40b77f158947a497e8a65a3e0d3378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } } }
I20260810 23:37:59.777869 13382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:59.780337 13406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260810 23:37:59.788674 13014 tablet.cc:2369] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:37:59.815939 13424 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:37:59.817935 13406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:37:59.823434 13212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } attempt: 5
Aug 10 23:37:59 dist-test-slave-04bb krb5kdc[12953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405079, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.253@KRBTEST.COM for kudu/127.9.48.252@KRBTEST.COM
I20260810 23:37:59.842252 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.781073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.9.48.253:43953)
0810 23:37:59.781281 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:59.781288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:59.792206 (+ 10918us) server_negotiation.cc:407] Connection header received
0810 23:37:59.794043 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:59.794049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:59.794167 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:59.794242 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:59.800111 (+ 5869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.801403 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.812721 (+ 11318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.813475 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.813584 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.827739 (+ 14155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:59.827748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:59.827756 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:59.827815 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:59.836836 (+ 9021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.837675 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.837681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.837685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.837839 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.839824 (+ 1985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.839829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.839835 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.840204 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:59.841491 (+ 1287us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:59.841624 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 23:37:59.841856 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:37:59.846163 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.782000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.9.48.252:43229)
0810 23:37:59.782235 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:59.782244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:59.793630 (+ 11386us) server_negotiation.cc:407] Connection header received
0810 23:37:59.793794 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:59.793800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:59.793912 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:59.794017 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:59.797729 (+ 3712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.798926 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.805303 (+ 6377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.805954 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.806092 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.825214 (+ 19122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:59.825221 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:59.825226 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:59.825267 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:59.842276 (+ 17009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.843156 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.843165 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.843170 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.843322 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.843792 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.843797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.843801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.844178 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:59.844412 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:59.845543 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0810 23:37:59.845799 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:37:59.818979 13224 master_runner.cc:186] Error getting master registration for 127.9.48.254:41411: OK, instance_id { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" instance_seqno: 1786405078548935 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:37:59.850543 13224 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:37:59.852593 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.785682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52350 (local address 127.9.48.254:41411)
0810 23:37:59.785899 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:59.785907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:59.815523 (+ 29616us) server_negotiation.cc:407] Connection header received
0810 23:37:59.815686 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:59.815692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:59.815785 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:59.815905 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:59.820620 (+ 4715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.821871 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.833622 (+ 11751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.834477 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.834614 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.838512 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:59.838522 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:59.838534 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:59.838608 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:59.843946 (+ 5338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.845883 (+ 1937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.845891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.845897 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.846089 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.846961 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.846969 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.846975 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.847462 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:59.847798 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:59.849224 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0810 23:37:59.849460 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:37:59.858287 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.824060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:43229 (local address 127.0.0.1:60382)
0810 23:37:59.824451 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 23:37:59.824474 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:37:59.824607 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:37:59.827719 (+ 3112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:37:59.827724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:37:59.828909 (+ 1185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:37:59.829149 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.829165 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.832328 (+ 3163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.832335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.833805 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:37:59.833817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.837048 (+ 3231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.837056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:37:59.837319 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.838033 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:37:59.838058 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:37:59.848694 (+ 10636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:37:59.854624 (+ 5930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.854632 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.854638 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.855346 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.855754 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:37:59.855762 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:37:59.855769 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:37:59.855951 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:37:59.857237 (+ 1286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:37:59.857244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:37:59.857414 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 23:37:59.857568 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 23:37:59.857885 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253}
I20260810 23:37:59.859251 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:37:59.824843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60382 (local address 127.9.48.252:43229)
0810 23:37:59.825178 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0810 23:37:59.825189 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:37:59.825208 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:37:59.825259 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:37:59.825264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:37:59.825357 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:37:59.825468 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:37:59.829344 (+ 3876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.830478 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.833982 (+ 3504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:37:59.834719 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:37:59.834858 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:37:59.849832 (+ 14974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:37:59.849842 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:37:59.849850 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:37:59.849898 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:37:59.854478 (+ 4580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.855476 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.855481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.855485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.855637 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:37:59.856070 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:37:59.856075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:37:59.856079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:37:59.856462 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:37:59.857159 (+ 697us) server_negotiation.cc:1035] Waiting for connection context
0810 23:37:59.858669 (+ 1510us) server_negotiation.cc:299] Negotiation successful
0810 23:37:59.858906 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260810 23:37:59.865420 13304 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d [term 1 FOLLOWER]: Refusing update from remote peer 7b40b77f158947a497e8a65a3e0d3378: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:37:59.866482 13360 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:37:59.866677 13212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d: Bootstrap starting.
I20260810 23:37:59.871922 13212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d: Neither blocks nor log segments found. Creating new log.
I20260810 23:37:59.873493 13212 log.cc:824] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d: Log is configured to *not* fsync() on all Append() calls
W20260810 23:37:59.876500 13267 tablet.cc:2369] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:37:59.881839 13212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d: No bootstrap required, opened a new log
I20260810 23:37:59.837387 12979 master_runner.cc:186] Error getting master registration for 127.9.48.253:43953: OK, instance_id { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" instance_seqno: 1786405079094975 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:37:59.889268 12979 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:37:59.896174 13212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } }
I20260810 23:37:59.896598 13212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46a12ee77ff94e2da8d9e95a44d13f9d, State: Initialized, Role: FOLLOWER
I20260810 23:37:59.897150 13212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } }
I20260810 23:37:59.898381 13359 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b40b77f158947a497e8a65a3e0d3378. Latest consensus state: current_term: 1 leader_uuid: "7b40b77f158947a497e8a65a3e0d3378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } } }
I20260810 23:37:59.898727 13359 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:37:59.900484 13432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } } }
I20260810 23:37:59.900965 13432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:37:59.902124 13212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:37:59.903158 13162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:37:59.908066 13162 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d [term 1 FOLLOWER]: Refusing update from remote peer 7b40b77f158947a497e8a65a3e0d3378: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:37:59.908661 13427 mvcc.cc:205] Tried to move back new op lower bound from 7317115207118016512 to 7317115206696927232. Current Snapshot: MvccSnapshot[applied={T|T < 7317115207118016512}]
I20260810 23:37:59.909838 13429 mvcc.cc:205] Tried to move back new op lower bound from 7317115207118016512 to 7317115206696927232. Current Snapshot: MvccSnapshot[applied={T|T < 7317115207118016512}]
I20260810 23:37:59.911230 13406 catalog_manager.cc:1347] Generated new cluster ID: cea8e807e6324df8a0d10c15bdcb05d8
I20260810 23:37:59.911320 13406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:37:59.912321 13163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7b40b77f158947a497e8a65a3e0d3378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d"
I20260810 23:37:59.916114 13360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b40b77f158947a497e8a65a3e0d3378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } } }
I20260810 23:37:59.916494 13360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:59.923358 13163 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b40b77f158947a497e8a65a3e0d3378 in term 1.
I20260810 23:37:59.925006 13382 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:37:59.932345 13359 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b40b77f158947a497e8a65a3e0d3378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } } }
I20260810 23:37:59.932655 13359 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:37:59.937741 13406 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:37:59.939628 13406 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:37:59.968310 13406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378: Generated new TSK 0
I20260810 23:37:59.969444 13406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:37:59.969597 13382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b40b77f158947a497e8a65a3e0d3378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } } }
I20260810 23:37:59.969931 13382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b40b77f158947a497e8a65a3e0d3378 [sys.catalog]: This master's current role is: LEADER
I20260810 23:37:59.969743 13432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b40b77f158947a497e8a65a3e0d3378. Latest consensus state: current_term: 1 leader_uuid: "7b40b77f158947a497e8a65a3e0d3378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } } }
I20260810 23:37:59.970096 13432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:37:59.969770 13456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b40b77f158947a497e8a65a3e0d3378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b40b77f158947a497e8a65a3e0d3378" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 41411 } } peers { permanent_uuid: "46a12ee77ff94e2da8d9e95a44d13f9d" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 43953 } } peers { permanent_uuid: "b7c44e77e706448881ba441bb244aa5d" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 43229 } } }
I20260810 23:37:59.970223 13456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:37:59.996351 13469 catalog_manager.cc:1259] Loaded cluster ID: cea8e807e6324df8a0d10c15bdcb05d8
I20260810 23:37:59.996459 13469 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d: loading cluster ID for follower catalog manager: success
I20260810 23:38:00.000509 13469 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d: acquiring CA information for follower catalog manager: success
I20260810 23:38:00.003165 13469 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 46a12ee77ff94e2da8d9e95a44d13f9d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 23:38:00.079902 13220 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:00.069020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40892 (local address 127.9.48.253:43953)
0810 23:38:00.069280 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.069289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.077177 (+ 7888us) server_negotiation.cc:407] Connection header received
0810 23:38:00.077537 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.077547 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.077701 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.077822 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:00.079534 (+ 1712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40892: BlockingRecv error: recv got EOF from 127.0.0.1:40892 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260810 23:38:00.082377 13350 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:00.068509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.9.48.254:41411)
0810 23:38:00.069073 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.069081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.080729 (+ 11648us) server_negotiation.cc:407] Connection header received
0810 23:38:00.080992 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.080997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.081091 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.081197 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:00.082102 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52354: BlockingRecv error: recv got EOF from 127.0.0.1:52354 (error 108)
Metrics: {"server-negotiator.queue_time_us":409}
W20260810 23:38:00.083667 13347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:00.070632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60386 (local address 127.9.48.252:43229)
0810 23:38:00.070811 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.070817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.080965 (+ 10148us) server_negotiation.cc:407] Connection header received
0810 23:38:00.081307 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.081317 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.081431 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.081523 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:00.083351 (+ 1828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60386: BlockingRecv error: recv got EOF from 127.0.0.1:60386 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:38:00.133068 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:00.110270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40894 (local address 127.9.48.253:43953)
0810 23:38:00.110468 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.110475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.125083 (+ 14608us) server_negotiation.cc:407] Connection header received
0810 23:38:00.125456 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.125462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.125677 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.125816 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:00.126550 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.127766 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.129906 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.130651 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.130753 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:00.131508 (+ 755us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:00.132200 (+ 692us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:00.132321 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:00.132599 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 23:38:00.132696 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 23:38:00.147557 13347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:00.142019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.9.48.252:43229)
0810 23:38:00.142207 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.142214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.145939 (+ 3725us) server_negotiation.cc:407] Connection header received
0810 23:38:00.146162 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.146170 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.146318 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.146429 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:00.147271 (+ 842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60392: BlockingRecv error: recv got EOF from 127.0.0.1:60392 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:38:00.163527 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:00.151656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.9.48.253:43953)
0810 23:38:00.151845 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.151852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.157070 (+ 5218us) server_negotiation.cc:407] Connection header received
0810 23:38:00.157259 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.157271 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.157366 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.157477 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:00.157990 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.159191 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.161141 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.161910 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.162029 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:00.162612 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:00.162760 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:00.162854 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:00.163072 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 23:38:00.163178 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 23:38:00.178826 13347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:00.171719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.9.48.252:43229)
0810 23:38:00.171907 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.171913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.177074 (+ 5161us) server_negotiation.cc:407] Connection header received
0810 23:38:00.177381 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.177388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.177496 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.177605 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:00.178589 (+ 984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60404: BlockingRecv error: recv got EOF from 127.0.0.1:60404 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:38:00.191831 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:00.184670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.9.48.253:43953)
0810 23:38:00.184849 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.184855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.185623 (+ 768us) server_negotiation.cc:407] Connection header received
0810 23:38:00.185812 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.185819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.185906 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.186010 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:00.186509 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.187680 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.189642 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.190312 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.190442 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:00.190933 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:00.191111 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:00.191201 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:00.191391 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 23:38:00.191487 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 23:38:00.206557 13347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:00.199360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60416 (local address 127.9.48.252:43229)
0810 23:38:00.199553 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.199560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.205047 (+ 5487us) server_negotiation.cc:407] Connection header received
0810 23:38:00.205268 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.205274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.205373 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.205477 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:00.206314 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60416: BlockingRecv error: recv got EOF from 127.0.0.1:60416 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:38:00.225567 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:00.218483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.9.48.253:43953)
0810 23:38:00.218665 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.218671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.219173 (+ 502us) server_negotiation.cc:407] Connection header received
0810 23:38:00.219356 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.219368 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.219471 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.219573 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:00.220055 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.221300 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.223367 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.224102 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.224254 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:00.224763 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:00.224907 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:00.225008 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:00.225171 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 23:38:00.225270 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 23:38:00.244750 13347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:00.242381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.9.48.252:43229)
0810 23:38:00.242660 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.242668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.243174 (+ 506us) server_negotiation.cc:407] Connection header received
0810 23:38:00.243386 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.243392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.243489 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.243595 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:00.244474 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60428: BlockingRecv error: recv got EOF from 127.0.0.1:60428 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
I20260810 23:38:00.264412 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:00.256417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.9.48.253:43953)
0810 23:38:00.256654 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.256660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.257189 (+ 529us) server_negotiation.cc:407] Connection header received
0810 23:38:00.257411 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.257417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.257515 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.257614 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:00.258092 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.259274 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.261321 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.262178 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.262328 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:00.262988 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:00.263153 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:00.263251 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:00.263940 (+ 689us) server_negotiation.cc:299] Negotiation successful
0810 23:38:00.264042 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 23:38:00.274515 13347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:00.271920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60440 (local address 127.9.48.252:43229)
0810 23:38:00.272135 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.272142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.272952 (+ 810us) server_negotiation.cc:407] Connection header received
0810 23:38:00.273135 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.273144 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.273243 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.273353 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:00.274257 (+ 904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60440: BlockingRecv error: recv got EOF from 127.0.0.1:60440 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:38:00.303663 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:00.294365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.9.48.253:43953)
0810 23:38:00.296271 (+ 1906us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.296279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.296898 (+ 619us) server_negotiation.cc:407] Connection header received
0810 23:38:00.297117 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.297123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.297213 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.297328 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:00.297829 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.299093 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.301242 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.301925 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.302058 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:00.302729 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:00.302933 (+ 204us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:00.303039 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:00.303232 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 23:38:00.303337 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1742}
W20260810 23:38:00.312695 13347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:00.310594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.9.48.252:43229)
0810 23:38:00.310791 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.310798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.311022 (+ 224us) server_negotiation.cc:407] Connection header received
0810 23:38:00.311229 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.311235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.311336 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.311454 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:00.312416 (+ 962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60452: BlockingRecv error: recv got EOF from 127.0.0.1:60452 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:38:00.358491 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:00.350665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.9.48.253:43953)
0810 23:38:00.350892 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.350898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.351458 (+ 560us) server_negotiation.cc:407] Connection header received
0810 23:38:00.351697 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.351704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.351822 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.351944 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:00.352491 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.353707 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.356142 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.356902 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.357009 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:00.357630 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:00.357766 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:00.357867 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:00.358044 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 23:38:00.358130 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260810 23:38:00.367558 13347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:00.365614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.9.48.252:43229)
0810 23:38:00.365798 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.365804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.365868 (+ 64us) server_negotiation.cc:407] Connection header received
0810 23:38:00.366145 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.366151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.366254 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.366357 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:00.367310 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60464: BlockingRecv error: recv got EOF from 127.0.0.1:60464 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:38:00.442780 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:00.436073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.9.48.253:43953)
0810 23:38:00.436285 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.436292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.436714 (+ 422us) server_negotiation.cc:407] Connection header received
0810 23:38:00.436899 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.436906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.437005 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.437137 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:00.437554 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.438820 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.440664 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.441453 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.441600 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:00.442003 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:00.442140 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:00.442252 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:00.442406 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 23:38:00.442493 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 23:38:00.450485 13347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:00.448620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.9.48.252:43229)
0810 23:38:00.448826 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.448832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.449178 (+ 346us) server_negotiation.cc:407] Connection header received
0810 23:38:00.449334 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.449341 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.449455 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.449563 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:00.450268 (+ 705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60478: BlockingRecv error: recv got EOF from 127.0.0.1:60478 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:38:00.589118 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:00.583911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.9.48.253:43953)
0810 23:38:00.584160 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.584168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.584570 (+ 402us) server_negotiation.cc:407] Connection header received
0810 23:38:00.584811 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.584816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.584884 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.584983 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:00.585403 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.586095 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.587424 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.587850 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.587986 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:00.588397 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:00.588497 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:00.588623 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:00.588763 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 23:38:00.588850 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 23:38:00.596920 13347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:00.594826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60494 (local address 127.9.48.252:43229)
0810 23:38:00.595059 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.595065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.595351 (+ 286us) server_negotiation.cc:407] Connection header received
0810 23:38:00.595548 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.595558 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.595632 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.595734 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:00.596698 (+ 964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60494: BlockingRecv error: recv got EOF from 127.0.0.1:60494 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:38:00.820503 13424 catalog_manager.cc:1259] Loaded cluster ID: cea8e807e6324df8a0d10c15bdcb05d8
I20260810 23:38:00.820652 13424 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d: loading cluster ID for follower catalog manager: success
I20260810 23:38:00.823282 13424 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d: acquiring CA information for follower catalog manager: success
I20260810 23:38:00.825145 13424 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7c44e77e706448881ba441bb244aa5d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:38:00.862314 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:00.856742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40974 (local address 127.9.48.253:43953)
0810 23:38:00.856970 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.856977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.857253 (+ 276us) server_negotiation.cc:407] Connection header received
0810 23:38:00.857501 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.857511 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.857598 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.857699 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:00.858158 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.858901 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.860260 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.860808 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.860939 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:00.861444 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:00.861562 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:00.861614 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:00.861853 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 23:38:00.861970 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 23:38:00.875644 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:00.869556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.9.48.252:43229)
0810 23:38:00.869794 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.869800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.870170 (+ 370us) server_negotiation.cc:407] Connection header received
0810 23:38:00.870348 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.870352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.870490 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.870586 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:00.871062 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.871839 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.873240 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.873823 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.873984 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:00.874496 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:00.875031 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:00.875198 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:00.875250 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 23:38:00.875347 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:38:00.900928 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:00.882636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40982 (local address 127.9.48.253:43953)
0810 23:38:00.882848 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.882855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.892043 (+ 9188us) server_negotiation.cc:407] Connection header received
0810 23:38:00.893095 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.893102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.893197 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.893263 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:00.894681 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.895844 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.898436 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.899074 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.899170 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:00.900003 (+ 833us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:00.900146 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:00.900208 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:00.900447 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 23:38:00.900568 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:38:00.903601 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:00.882663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60512 (local address 127.9.48.252:43229)
0810 23:38:00.882880 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.882888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.895171 (+ 12283us) server_negotiation.cc:407] Connection header received
0810 23:38:00.896193 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.896198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.896291 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.896363 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:00.897510 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.898693 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.901186 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.901898 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.901990 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:00.902698 (+ 708us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:00.902854 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:00.902930 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:00.903166 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 23:38:00.903249 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:38:00.904284 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:00.882299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.9.48.254:41411)
0810 23:38:00.894370 (+ 12071us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:00.894380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:00.895852 (+ 1472us) server_negotiation.cc:407] Connection header received
0810 23:38:00.896038 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:00.896047 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:00.896287 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:00.896470 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:00.897528 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.898646 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.901117 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:00.901752 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:00.901885 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:00.902801 (+ 916us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:00.903324 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:00.903425 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:00.903916 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 23:38:00.904017 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11878,"thread_start_us":12082,"threads_started":1}
I20260810 23:38:00.910517 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 12979
I20260810 23:38:00.911754 13075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:01.039858 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 12979
I20260810 23:38:01.074136 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 13097
I20260810 23:38:01.075412 13201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:01.184947 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 13097
I20260810 23:38:01.216456 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 13224
I20260810 23:38:01.217597 13336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:01.326650 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 13224
2026-08-10T23:38:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4936 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:38:01 dist-test-slave-04bb krb5kdc[13670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:38:01 dist-test-slave-04bb krb5kdc[13670](info): set up 2 sockets
Aug 10 23:38:01 dist-test-slave-04bb krb5kdc[13670](info): commencing operation
krb5kdc: starting...
W20260810 23:38:02.715135 9411 mini_kdc.cc:121] Time spent starting KDC: real 1.337s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/test-user.keytab.
Aug 10 23:38:02 dist-test-slave-04bb krb5kdc[13670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:38:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:38:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.48.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
Entry for principal kudu/127.9.48.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab.
I20260810 23:38:03.055542 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:43695
--webserver_interface=127.9.48.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:43695,127.9.48.253:37477,127.9.48.252:33209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:38:03.405035 13692 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:38:03.405387 13692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:03.405524 13692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:03.405650 13692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:03.412825 13692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:03.412906 13692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:03.413020 13692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:03.413100 13692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:03.413196 13692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:38:03.422786 13692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:43695,127.9.48.253:37477,127.9.48.252:33209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.254.keytab
--principal=kudu/127.9.48.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.254:43695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.48.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:03.424155 13692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:03.425539 13692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:03.437188 13706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:03.437273 13703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:03.437250 13704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:03.442003 13692 server_base.cc:1034] running on GCE node
I20260810 23:38:03.442777 13692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:03.444186 13692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:03.445429 13692 hybrid_clock.cc:648] HybridClock initialized: now 1786405083445364 us; error 40 us; skew 500 ppm
I20260810 23:38:03.445770 13692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:03 dist-test-slave-04bb krb5kdc[13670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405083, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:03.449083 13692 init.cc:377] Logged in from keytab as kudu/127.9.48.254@KRBTEST.COM (short username kudu)
I20260810 23:38:03.450635 13692 webserver.cc:460] Webserver started at http://127.9.48.254:39985/ using document root <none> and password file <none>
I20260810 23:38:03.451355 13692 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:03.451486 13692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:03.451807 13692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:03.454861 13692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/data/instance:
uuid: "40b0fe0899fe4b61b2574f7ec4191446"
format_stamp: "Formatted at 2026-08-10 23:38:03 on dist-test-slave-04bb"
server_key: "9c8d78323ac2877ed16537aa9a1243a4"
server_key_iv: "89e20c14b0550ec9fd9878c9da63958b"
server_key_version: "encryptionkey@0"
I20260810 23:38:03.455691 13692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/wal/instance:
uuid: "40b0fe0899fe4b61b2574f7ec4191446"
format_stamp: "Formatted at 2026-08-10 23:38:03 on dist-test-slave-04bb"
server_key: "9c8d78323ac2877ed16537aa9a1243a4"
server_key_iv: "89e20c14b0550ec9fd9878c9da63958b"
server_key_version: "encryptionkey@0"
I20260810 23:38:03.460911 13692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260810 23:38:03.464287 13716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:03.465638 13692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:38:03.465821 13692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/wal
uuid: "40b0fe0899fe4b61b2574f7ec4191446"
format_stamp: "Formatted at 2026-08-10 23:38:03 on dist-test-slave-04bb"
server_key: "9c8d78323ac2877ed16537aa9a1243a4"
server_key_iv: "89e20c14b0550ec9fd9878c9da63958b"
server_key_version: "encryptionkey@0"
I20260810 23:38:03.466049 13692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:03.488457 13692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:03.492936 13692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:03.493202 13692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:03.512136 13692 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.254:43695
I20260810 23:38:03.512174 13796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.254:43695 every 8 connection(s)
I20260810 23:38:03.513535 13692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/data/info.pb
I20260810 23:38:03.517278 13797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:03.519731 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 13692
I20260810 23:38:03.519942 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-0/wal/instance
I20260810 23:38:03.520551 9411 external_mini_cluster.cc:1428] Setting key b6a7521810e8ad54fb4f1d80b038698e
I20260810 23:38:03.526644 13797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:38:03 dist-test-slave-04bb krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405083, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:38:03.551730 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:03.530804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37800 (local address 127.9.48.254:43695)
0810 23:38:03.533053 (+ 2249us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:03.533060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:03.533118 (+ 58us) server_negotiation.cc:407] Connection header received
0810 23:38:03.535274 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:03.535280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:03.535526 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:03.535695 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:03.537351 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:03.538959 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:03.540318 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:03.540919 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:03.541149 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:03.544426 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:03.544433 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:03.544445 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:03.544478 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:03.547303 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:03.547973 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:03.547981 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:03.547985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:03.548132 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:03.548514 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:03.548521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:03.548545 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:03.549039 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:03.549252 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:03.549812 (+ 560us) server_negotiation.cc:299] Negotiation successful
0810 23:38:03.550189 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":2024,"spinlock_wait_cycles":25984,"thread_start_us":1734,"threads_started":1}
I20260810 23:38:03.551751 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:03.530804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:43695 (local address 127.0.0.1:37800)
0810 23:38:03.531806 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:03.531873 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:03.534875 (+ 3002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:03.535771 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:03.535781 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:03.536647 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:03.537097 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:03.537115 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:03.539103 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:03.539106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:03.540137 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:03.540149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:03.541193 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:03.541200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:03.541391 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:03.542048 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:03.542070 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:03.544208 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:03.547426 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:03.547440 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:03.547450 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:03.547849 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:03.548243 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:03.548249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:03.548253 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:03.548411 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:03.549194 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:03.549203 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:03.549419 (+ 216us) client_negotiation.cc:769] Sending connection context
0810 23:38:03.549666 (+ 247us) client_negotiation.cc:240] Negotiation successful
0810 23:38:03.550315 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":156,"thread_start_us":439,"threads_started":1}
I20260810 23:38:03.555087 13797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.48.253@KRBTEST.COM; defaulting to no policy
I20260810 23:38:03.556119 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:03.555703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:37477 (local address 127.0.0.1:53032)
0810 23:38:03.555863 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:03.555996 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 23:38:03.556429 13721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
I20260810 23:38:03.557924 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:03.557567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:37477 (local address 127.0.0.1:53044)
0810 23:38:03.557734 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:03.557805 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:38:03.558157 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.253:37477: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.253.keytab.
Entry for principal kudu/127.9.48.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.253.keytab.
I20260810 23:38:03.596222 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.253:37477
--webserver_interface=127.9.48.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.253.keytab
--principal=kudu/127.9.48.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:43695,127.9.48.253:37477,127.9.48.252:33209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:38:03.607554 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } attempt: 1
I20260810 23:38:03.608867 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:03.608372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:37477 (local address 127.0.0.1:53054)
0810 23:38:03.608614 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:03.608719 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 23:38:03.609993 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:03.609535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:37477 (local address 127.0.0.1:53062)
0810 23:38:03.609729 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:03.609821 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 23:38:03.610345 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.253:37477: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
I20260810 23:38:03.678591 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } attempt: 2
I20260810 23:38:03.679860 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:03.679263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:37477 (local address 127.0.0.1:53074)
0810 23:38:03.679619 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:03.679712 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 23:38:03.680954 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:03.680605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:37477 (local address 127.0.0.1:53080)
0810 23:38:03.680747 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:03.680829 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:38:03.681123 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.253:37477: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
I20260810 23:38:03.772367 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } attempt: 3
I20260810 23:38:03.773669 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:03.773122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:37477 (local address 127.0.0.1:53088)
0810 23:38:03.773363 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:03.773482 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 23:38:03.774899 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:03.774455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:37477 (local address 127.0.0.1:53092)
0810 23:38:03.774617 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:03.774709 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 23:38:03.775117 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.253:37477: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
I20260810 23:38:03.918416 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } attempt: 4
I20260810 23:38:03.919786 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:03.919131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:37477 (local address 127.0.0.1:53098)
0810 23:38:03.919485 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:03.919596 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260810 23:38:03.921072 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:03.920645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:37477 (local address 127.0.0.1:53114)
0810 23:38:03.920815 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:03.920917 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 23:38:03.921345 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.253:37477: Network error: Client connection negotiation failed: client connection to 127.9.48.253:37477: connect: Connection refused (error 111)
W20260810 23:38:03.940214 13804 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:38:03.940609 13804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:03.940735 13804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:03.940874 13804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:03.948261 13804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:03.948374 13804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:03.948494 13804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:03.948598 13804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:03.948697 13804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:38:03.958591 13804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:43695,127.9.48.253:37477,127.9.48.252:33209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.253.keytab
--principal=kudu/127.9.48.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.253:37477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.48.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:03.959888 13804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:03.961346 13804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:03.973071 13813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:03.972941 13820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:03.973107 13815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:03.974546 13804 server_base.cc:1034] running on GCE node
I20260810 23:38:03.975379 13804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:03.976853 13804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:03.978097 13804 hybrid_clock.cc:648] HybridClock initialized: now 1786405083978032 us; error 34 us; skew 500 ppm
I20260810 23:38:03.978430 13804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:03 dist-test-slave-04bb krb5kdc[13670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405083, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:03.982003 13804 init.cc:377] Logged in from keytab as kudu/127.9.48.253@KRBTEST.COM (short username kudu)
I20260810 23:38:03.983657 13804 webserver.cc:460] Webserver started at http://127.9.48.253:33475/ using document root <none> and password file <none>
I20260810 23:38:03.984416 13804 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:03.984583 13804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:03.984939 13804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:03.988318 13804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/data/instance:
uuid: "472b634036a94f888efc4a74f0ebb061"
format_stamp: "Formatted at 2026-08-10 23:38:03 on dist-test-slave-04bb"
server_key: "c4f96003101388cb4c4f9986346b40f5"
server_key_iv: "196334c8a559b275c8fb17dad78d0686"
server_key_version: "encryptionkey@0"
I20260810 23:38:03.989308 13804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/wal/instance:
uuid: "472b634036a94f888efc4a74f0ebb061"
format_stamp: "Formatted at 2026-08-10 23:38:03 on dist-test-slave-04bb"
server_key: "c4f96003101388cb4c4f9986346b40f5"
server_key_iv: "196334c8a559b275c8fb17dad78d0686"
server_key_version: "encryptionkey@0"
I20260810 23:38:03.994972 13804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:38:03.998826 13834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:04.000231 13804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:38:04.000432 13804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/wal
uuid: "472b634036a94f888efc4a74f0ebb061"
format_stamp: "Formatted at 2026-08-10 23:38:03 on dist-test-slave-04bb"
server_key: "c4f96003101388cb4c4f9986346b40f5"
server_key_iv: "196334c8a559b275c8fb17dad78d0686"
server_key_version: "encryptionkey@0"
I20260810 23:38:04.000751 13804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:04.017354 13804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:04.021844 13804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:04.022171 13804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:04.042621 13804 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.253:37477
I20260810 23:38:04.042631 13919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.253:37477 every 8 connection(s)
I20260810 23:38:04.044327 13804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/data/info.pb
I20260810 23:38:04.048161 13921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:04.049799 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 13804
I20260810 23:38:04.050002 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-1/wal/instance
I20260810 23:38:04.050496 9411 external_mini_cluster.cc:1428] Setting key eed34a293a39a2e16665b3ac1e416adf
I20260810 23:38:04.055114 13921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:38:04 dist-test-slave-04bb krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405083, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.253@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:38:04.079085 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.057824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.9.48.254:43695)
0810 23:38:04.058389 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:04.058396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:04.059333 (+ 937us) server_negotiation.cc:407] Connection header received
0810 23:38:04.061333 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:04.061342 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:04.061441 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:04.061571 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:04.063081 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.064311 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.065897 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.066320 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.066421 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.071114 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:04.071125 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:04.071133 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:04.071213 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:04.075618 (+ 4405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.076372 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.076381 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.076387 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.076573 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.077019 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.077028 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.077033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.077459 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:04.077621 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:04.078283 (+ 662us) server_negotiation.cc:299] Negotiation successful
0810 23:38:04.078555 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":276,"threads_started":1}
I20260810 23:38:04.079696 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.058289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:43695 (local address 127.0.0.1:37816)
0810 23:38:04.059134 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.059208 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:04.061059 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:04.061778 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:04.061785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:04.062309 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:04.062866 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.062883 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.064459 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.064463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.065729 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.065748 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.066567 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.066574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.066899 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.067648 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:04.067689 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:04.070783 (+ 3094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:04.075776 (+ 4993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.075783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.075797 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.076221 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.076702 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.076705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.076708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.076899 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.077593 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:04.077599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:04.077916 (+ 317us) client_negotiation.cc:769] Sending connection context
0810 23:38:04.078137 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 23:38:04.078530 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":338,"threads_started":1}
I20260810 23:38:04.081655 13921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.48.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.48.252@KRBTEST.COM" created.
Aug 10 23:38:04 dist-test-slave-04bb krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405083, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.253@KRBTEST.COM for kudu/127.9.48.253@KRBTEST.COM
I20260810 23:38:04.102669 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.082488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:37477 (local address 127.0.0.1:53116)
0810 23:38:04.082697 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.082720 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:04.082864 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:04.083784 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:04.083796 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:04.084332 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:04.084567 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.084583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.085977 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.085985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.087483 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.087499 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.088386 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.088398 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.088612 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.089298 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:04.089316 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:04.093799 (+ 4483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:04.099184 (+ 5385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.099225 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.099230 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.099725 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.100287 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.100293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.100297 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.100518 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.101508 (+ 990us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:04.101516 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:04.101705 (+ 189us) client_negotiation.cc:769] Sending connection context
0810 23:38:04.101868 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 23:38:04.102188 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 23:38:04.103245 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.082698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.9.48.253:37477)
0810 23:38:04.083374 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:04.083380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:04.083405 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:38:04.083447 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:04.083452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:04.083618 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:04.083730 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:04.084762 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.085807 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.087654 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.088211 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.088314 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.094157 (+ 5843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:04.094175 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:04.094198 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:04.094251 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:04.099024 (+ 4773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.099882 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.099895 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.099900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.100091 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.100724 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.100729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.100733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.101311 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:04.101563 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:04.102181 (+ 618us) server_negotiation.cc:299] Negotiation successful
0810 23:38:04.102511 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":395,"threads_started":1}
I20260810 23:38:04.106217 13921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:38:04.107522 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.106962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40816)
0810 23:38:04.107177 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.107316 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 23:38:04.107884 13840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
I20260810 23:38:04.109575 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.109211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40832)
0810 23:38:04.109341 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.109413 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:38:04.109822 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:33209: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.48.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.252.keytab.
Entry for principal kudu/127.9.48.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.252.keytab.
I20260810 23:38:04.133565 9411 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskahRJgu/build/asan/bin/kudu
/tmp/dist-test-taskahRJgu/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.252:33209
--webserver_interface=127.9.48.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.252.keytab
--principal=kudu/127.9.48.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.48.212:34877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.48.254:43695,127.9.48.253:37477,127.9.48.252:33209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:38:04.159178 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } attempt: 1
I20260810 23:38:04.160636 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.159983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40846)
0810 23:38:04.160243 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.160387 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 23:38:04.161897 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.161416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40858)
0810 23:38:04.161596 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.161719 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 23:38:04.162206 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:33209: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
I20260810 23:38:04.220651 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } attempt: 5
Aug 10 23:38:04 dist-test-slave-04bb krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405083, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for kudu/127.9.48.253@KRBTEST.COM
I20260810 23:38:04.230487 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } attempt: 2
I20260810 23:38:04.231623 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.231085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40874)
0810 23:38:04.231299 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.231401 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 23:38:04.232896 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.232416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40884)
0810 23:38:04.232620 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.232698 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:38:04.233271 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:33209: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
I20260810 23:38:04.236218 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.221341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:37477 (local address 127.0.0.1:53122)
0810 23:38:04.221637 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.221666 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:04.221841 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:04.222240 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:04.222247 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:04.222880 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:04.223128 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.223142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.224097 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.224103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.225486 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.225502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.226218 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.226224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.226449 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.227104 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:04.227127 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:04.229527 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:04.233765 (+ 4238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.233769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.233772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.234175 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.234646 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.234660 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.234662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.234814 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.235442 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:04.235446 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:04.235589 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 23:38:04.235758 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 23:38:04.235967 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 23:38:04.236400 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.221643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53122 (local address 127.9.48.253:37477)
0810 23:38:04.221813 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:04.221819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:04.221838 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:38:04.221978 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:04.221981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:04.222060 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:04.222144 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:04.223273 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.223960 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.225625 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.226069 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.226161 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.229807 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:04.229812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:04.229816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:04.229854 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:04.233621 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.234311 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.234320 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.234325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.234492 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.234939 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.234946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.234949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.235312 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:04.235465 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:04.235904 (+ 439us) server_negotiation.cc:299] Negotiation successful
0810 23:38:04.236118 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:38:04.237582 13797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:38:04.238674 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.238237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40890)
0810 23:38:04.238416 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.238508 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 23:38:04.239780 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.239335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40906)
0810 23:38:04.239516 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.239617 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 23:38:04.240037 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:33209: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
I20260810 23:38:04.291340 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } attempt: 1
I20260810 23:38:04.292719 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.292015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40918)
0810 23:38:04.292301 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.292444 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 23:38:04.294044 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.293635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40920)
0810 23:38:04.293815 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.293907 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 23:38:04.294281 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:33209: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
I20260810 23:38:04.324623 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } attempt: 3
I20260810 23:38:04.326045 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.325440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40928)
0810 23:38:04.325712 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.325847 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260810 23:38:04.327284 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.326796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40930)
0810 23:38:04.327052 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.327150 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260810 23:38:04.327538 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:33209: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
I20260810 23:38:04.362540 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } attempt: 2
I20260810 23:38:04.363986 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.363273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40942)
0810 23:38:04.363586 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.363741 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260810 23:38:04.365358 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.364860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40952)
0810 23:38:04.365049 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.365168 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 23:38:04.365643 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:33209: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
I20260810 23:38:04.470783 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } attempt: 4
I20260810 23:38:04.471889 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } attempt: 3
I20260810 23:38:04.472215 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.471494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40954)
0810 23:38:04.471834 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.471971 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260810 23:38:04.473038 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.472482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40964)
0810 23:38:04.472746 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.472826 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260810 23:38:04.473660 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.473108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40976)
0810 23:38:04.473366 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.473476 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260810 23:38:04.473954 13921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:33209: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
I20260810 23:38:04.474243 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.473795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:40988)
0810 23:38:04.474024 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.474119 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260810 23:38:04.474490 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.48.252:33209: Network error: Client connection negotiation failed: client connection to 127.9.48.252:33209: connect: Connection refused (error 111)
W20260810 23:38:04.511853 13930 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:38:04.512194 13930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:38:04.512316 13930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:38:04.512444 13930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:38:04.519881 13930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:38:04.519979 13930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:38:04.520134 13930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:38:04.520220 13930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:38:04.520334 13930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:38:04.530545 13930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.48.212:34877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.48.254:43695,127.9.48.253:37477,127.9.48.252:33209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/krb5kdc/kudu_127.9.48.252.keytab
--principal=kudu/127.9.48.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.48.252:33209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.48.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 22:01:27 UTC on ccycloud.quasar-vyrfms.root.comops.site
build id 4579041
ASAN enabled
I20260810 23:38:04.531864 13930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:38:04.533365 13930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:38:04.545893 13936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:04.545930 13937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:38:04.545872 13940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:38:04.547051 13930 server_base.cc:1034] running on GCE node
I20260810 23:38:04.547812 13930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:38:04.549260 13930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:38:04.550479 13930 hybrid_clock.cc:648] HybridClock initialized: now 1786405084550399 us; error 55 us; skew 500 ppm
I20260810 23:38:04.550817 13930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 23:38:04 dist-test-slave-04bb krb5kdc[13670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405084, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:38:04.554184 13930 init.cc:377] Logged in from keytab as kudu/127.9.48.252@KRBTEST.COM (short username kudu)
I20260810 23:38:04.555534 13930 webserver.cc:460] Webserver started at http://127.9.48.252:44283/ using document root <none> and password file <none>
I20260810 23:38:04.556205 13930 fs_manager.cc:359] Metadata directory not provided
I20260810 23:38:04.556294 13930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:38:04.556624 13930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:38:04.559578 13930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/data/instance:
uuid: "e5f08dfe3fe045919387011ea36cf170"
format_stamp: "Formatted at 2026-08-10 23:38:04 on dist-test-slave-04bb"
server_key: "13eae50410098d454b1be961b46c33d4"
server_key_iv: "2782d1cc8ff687327c9b5038b54ba4f7"
server_key_version: "encryptionkey@0"
I20260810 23:38:04.560346 13930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/wal/instance:
uuid: "e5f08dfe3fe045919387011ea36cf170"
format_stamp: "Formatted at 2026-08-10 23:38:04 on dist-test-slave-04bb"
server_key: "13eae50410098d454b1be961b46c33d4"
server_key_iv: "2782d1cc8ff687327c9b5038b54ba4f7"
server_key_version: "encryptionkey@0"
I20260810 23:38:04.565526 13930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:38:04.568982 13954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:38:04.570362 13930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:38:04.570528 13930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/data,/tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/wal
uuid: "e5f08dfe3fe045919387011ea36cf170"
format_stamp: "Formatted at 2026-08-10 23:38:04 on dist-test-slave-04bb"
server_key: "13eae50410098d454b1be961b46c33d4"
server_key_iv: "2782d1cc8ff687327c9b5038b54ba4f7"
server_key_version: "encryptionkey@0"
I20260810 23:38:04.570756 13930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:38:04.594539 13930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:38:04.598954 13930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:38:04.599232 13930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:38:04.618096 13930 rpc_server.cc:307] RPC server started. Bound to: 127.9.48.252:33209
I20260810 23:38:04.618141 14035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.48.252:33209 every 8 connection(s)
I20260810 23:38:04.619704 13930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/data/info.pb
I20260810 23:38:04.623545 14037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:38:04.627585 9411 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskahRJgu/build/asan/bin/kudu as pid 13930
I20260810 23:38:04.627801 9411 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskahRJgu/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786404698557218-9411-0/minicluster-data/master-2/wal/instance
I20260810 23:38:04.628368 9411 external_mini_cluster.cc:1428] Setting key 39c0cf2e3a23a76f6131c34b9e4619fe
I20260810 23:38:04.631448 14037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:38:04 dist-test-slave-04bb krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.252@KRBTEST.COM
Aug 10 23:38:04 dist-test-slave-04bb krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405084, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.252@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:38:04.651738 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } attempt: 4
I20260810 23:38:04.652693 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.635758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.9.48.254:43695)
0810 23:38:04.636346 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:04.636354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:04.636930 (+ 576us) server_negotiation.cc:407] Connection header received
0810 23:38:04.637121 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:04.637128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:04.637229 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:04.637345 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:04.638491 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.639500 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.641585 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.642276 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.642451 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.646208 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:04.646218 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:04.646226 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:04.646275 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:04.649380 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.650233 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.650238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.650240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.650402 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.650937 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.650943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.650948 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.651297 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:04.651471 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:04.651971 (+ 500us) server_negotiation.cc:299] Negotiation successful
0810 23:38:04.652222 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":262,"threads_started":1}
I20260810 23:38:04.653193 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.635795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.254:43695 (local address 127.0.0.1:37820)
0810 23:38:04.636750 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.636818 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:04.636978 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:04.637413 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:04.637418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:04.638061 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:04.638318 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.638327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.639735 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.639741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.641409 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.641422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.642502 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.642507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.642632 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.643267 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:04.643288 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:04.645981 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:04.649504 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.649514 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.649521 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.650076 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.650526 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.650533 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.650540 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.650699 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.651445 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:04.651452 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:04.651647 (+ 195us) client_negotiation.cc:769] Sending connection context
0810 23:38:04.651839 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 23:38:04.652127 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":543,"threads_started":1}
I20260810 23:38:04.653631 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.630248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40992 (local address 127.9.48.252:33209)
0810 23:38:04.631143 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:04.631155 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:04.631199 (+ 44us) server_negotiation.cc:407] Connection header received
0810 23:38:04.632855 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:04.632877 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:04.633268 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:04.633587 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:04.635635 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.637640 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.639257 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.640301 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.640490 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.645294 (+ 4804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:04.645305 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:04.645320 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:04.645364 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:04.648370 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.649202 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.649208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.649215 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.649379 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.649781 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.649787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.649790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.650432 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:04.650769 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:04.651219 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 23:38:04.651647 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":411,"threads_started":1}
I20260810 23:38:04.655224 14037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:38:04 dist-test-slave-04bb krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405083, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.254@KRBTEST.COM for kudu/127.9.48.252@KRBTEST.COM
Aug 10 23:38:04 dist-test-slave-04bb krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405084, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.252@KRBTEST.COM for kudu/127.9.48.253@KRBTEST.COM
Aug 10 23:38:04 dist-test-slave-04bb krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.254@KRBTEST.COM
I20260810 23:38:04.670635 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.652347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:41000)
0810 23:38:04.652760 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.652782 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:04.652884 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:04.654085 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:04.654093 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:04.654545 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:04.654745 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.654757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.656733 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.656739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.658274 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.658289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.659017 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.659023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.659240 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.659890 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:04.659913 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:04.663044 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:04.667360 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.667371 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.667378 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.667936 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.668450 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.668458 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.668464 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.668695 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.669626 (+ 931us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:04.669633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:04.669817 (+ 184us) client_negotiation.cc:769] Sending connection context
0810 23:38:04.669919 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:38:04.670164 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278}
I20260810 23:38:04.671787 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.653052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.9.48.252:33209)
0810 23:38:04.653732 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:04.653741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:04.653764 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:38:04.653828 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:04.653835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:04.653906 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:04.654032 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:04.654984 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.656564 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.658422 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.658860 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.659002 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.663318 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:04.663325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:04.663330 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:04.663373 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:04.666900 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.668154 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.668161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.668165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.668308 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.668813 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.668822 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.668827 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.669339 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:04.671042 (+ 1703us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:04.671167 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 23:38:04.671379 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":342,"threads_started":1}
I20260810 23:38:04.673017 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.656209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53130 (local address 127.9.48.253:37477)
0810 23:38:04.656422 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:04.656428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:04.656450 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:38:04.656724 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:04.656730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:04.656812 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:04.656925 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:04.657868 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.658895 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.660675 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.661340 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.661454 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.665583 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:04.665592 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:04.665599 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:04.665655 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:04.669566 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.670537 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.670544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.670551 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.670741 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.671216 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.671221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.671224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.671553 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:04.671698 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:04.672293 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 23:38:04.672729 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 23:38:04.673940 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.656059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.253:37477 (local address 127.0.0.1:53130)
0810 23:38:04.656215 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.656233 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:04.656578 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:04.656966 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:04.656971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:04.657494 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:04.657657 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.657668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.659089 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.659096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.660486 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.660504 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.661723 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.661729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.661954 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.662576 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:04.662608 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:04.665365 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:04.669701 (+ 4336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.669706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.669710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.670379 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.670895 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.670900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.670903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.671084 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.671794 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:04.671800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:04.671981 (+ 181us) client_negotiation.cc:769] Sending connection context
0810 23:38:04.673327 (+ 1346us) client_negotiation.cc:240] Negotiation successful
0810 23:38:04.673596 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 23:38:04.674576 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.657379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37836 (local address 127.9.48.254:43695)
0810 23:38:04.657516 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:04.657520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:04.657724 (+ 204us) server_negotiation.cc:407] Connection header received
0810 23:38:04.657908 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:04.657914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:04.658013 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:04.658107 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:04.659612 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.660351 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.661614 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.662460 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.662609 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.667986 (+ 5377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:04.667991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:04.667995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:04.668026 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:04.671346 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.672039 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.672044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.672047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.672215 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.672767 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.672772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.672775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.673150 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:04.673396 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:04.673914 (+ 518us) server_negotiation.cc:299] Negotiation successful
0810 23:38:04.674371 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:38:04.675127 14037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:38:04.677992 13797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446: Bootstrap starting.
I20260810 23:38:04.682541 13797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:04.683898 13797 log.cc:824] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:38:04 dist-test-slave-04bb krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405084, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.252@KRBTEST.COM for kudu/127.9.48.252@KRBTEST.COM
I20260810 23:38:04.687561 13797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446: No bootstrap required, opened a new log
Aug 10 23:38:04 dist-test-slave-04bb krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.48.253@KRBTEST.COM
I20260810 23:38:04.691658 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.677444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:41014)
0810 23:38:04.677751 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.677771 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:04.677884 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:04.678682 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:04.678686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:04.679055 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:04.679187 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.679196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.680126 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.680132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.681347 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.681362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.682230 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.682235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.682492 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.683157 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:04.683178 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:04.685716 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:04.688676 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.688681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.688685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.689181 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.689690 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.689697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.689701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.689878 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.690563 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:04.690568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:04.690711 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 23:38:04.690859 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 23:38:04.691200 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":47}
I20260810 23:38:04.692021 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.677503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.9.48.252:33209)
0810 23:38:04.677673 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:04.677678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:04.677828 (+ 150us) server_negotiation.cc:407] Connection header received
0810 23:38:04.678431 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:04.678437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:04.678552 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:04.678666 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:04.679321 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.680002 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.681509 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.682077 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.682222 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.685907 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:04.685913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:04.685918 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:04.685953 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:04.688480 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.689346 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.689368 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.689372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.689533 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.690032 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.690038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.690042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.690444 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:04.690600 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:04.690978 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 23:38:04.691456 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":215,"server-negotiator.queue_time_us":65}
I20260810 23:38:04.697063 13797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } }
I20260810 23:38:04.697368 13797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b0fe0899fe4b61b2574f7ec4191446, State: Initialized, Role: FOLLOWER
I20260810 23:38:04.697762 13797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } }
I20260810 23:38:04.698195 14037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170: Bootstrap starting.
I20260810 23:38:04.698736 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.678264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.9.48.253:37477)
0810 23:38:04.678442 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:04.678448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:04.678530 (+ 82us) server_negotiation.cc:407] Connection header received
0810 23:38:04.678732 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:04.678737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:04.678809 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:04.678875 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:04.680663 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.681741 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.683651 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.684407 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.685375 (+ 968us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.690362 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:04.690368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:04.690373 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:04.690411 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:04.694327 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.695223 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.695229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.695232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.695395 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.695875 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.695880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.695883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.696227 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:04.696489 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:04.698052 (+ 1563us) server_negotiation.cc:299] Negotiation successful
0810 23:38:04.698377 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:38:04.698693 14051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } } }
I20260810 23:38:04.699086 14051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:38:04.699922 13797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:04.703325 14037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:04.704573 9411 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:38:04.704937 14037 log.cc:824] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:04.707895 14037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170: No bootstrap required, opened a new log
I20260810 23:38:04.708642 13753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "40b0fe0899fe4b61b2574f7ec4191446"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37836
I20260810 23:38:04.709302 13753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 23:38:04.709497 13753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:04.713215 13753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } }
I20260810 23:38:04.714799 13753 leader_election.cc:288] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [CANDIDATE]: Term 1 election: Requested vote from peers 472b634036a94f888efc4a74f0ebb061 (127.9.48.253:37477), e5f08dfe3fe045919387011ea36cf170 (127.9.48.252:33209)
W20260810 23:38:04.714969 14075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:38:04.715108 14075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:38:04.718709 14037 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } }
I20260810 23:38:04.719131 14037 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f08dfe3fe045919387011ea36cf170, State: Initialized, Role: FOLLOWER
I20260810 23:38:04.719678 14037 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } }
I20260810 23:38:04.721175 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } } }
I20260810 23:38:04.722208 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:38:04.722538 14000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40b0fe0899fe4b61b2574f7ec4191446" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e5f08dfe3fe045919387011ea36cf170"
I20260810 23:38:04.723001 14000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:04.726279 14000 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40b0fe0899fe4b61b2574f7ec4191446 in term 1.
I20260810 23:38:04.722515 14037 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:04.727046 13724 leader_election.cc:302] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40b0fe0899fe4b61b2574f7ec4191446, e5f08dfe3fe045919387011ea36cf170; no voters:
I20260810 23:38:04.727584 14051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:38:04.729771 14051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [term 1 LEADER]: Becoming Leader. State: Replica: 40b0fe0899fe4b61b2574f7ec4191446, State: Running, Role: LEADER
I20260810 23:38:04.730465 14051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } }
I20260810 23:38:04.736127 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40b0fe0899fe4b61b2574f7ec4191446. Latest consensus state: current_term: 1 leader_uuid: "40b0fe0899fe4b61b2574f7ec4191446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } } }
I20260810 23:38:04.718335 13692 master_runner.cc:186] Error getting master registration for 127.9.48.253:37477: OK, instance_id { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" instance_seqno: 1786405084036519 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:38:04.736574 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:04.736863 13692 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:38:04.738158 14090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:38:04.743281 14090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260810 23:38:04.745990 13726 tablet.cc:2369] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:38:04.768610 14000 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170 [term 1 FOLLOWER]: Refusing update from remote peer 40b0fe0899fe4b61b2574f7ec4191446: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:04.771096 14085 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:04.773176 13921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } attempt: 5
Aug 10 23:38:04 dist-test-slave-04bb krb5kdc[13670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405083, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.48.253@KRBTEST.COM for kudu/127.9.48.252@KRBTEST.COM
I20260810 23:38:04.791482 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.770149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53150 (local address 127.9.48.253:37477)
0810 23:38:04.770344 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:04.770353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:04.771003 (+ 650us) server_negotiation.cc:407] Connection header received
0810 23:38:04.771313 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:04.771320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:04.771415 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:04.771499 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:04.774163 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.775276 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.777299 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.778027 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.778133 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.782313 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:04.782321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:04.782329 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:04.782387 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:04.786276 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.787946 (+ 1670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.787951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.787954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.788107 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.788993 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.788998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.789000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.789397 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:04.789635 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:04.790855 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0810 23:38:04.791085 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:38:04.792223 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.774315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.9.48.252:33209)
0810 23:38:04.774495 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:04.774501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:04.774522 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:38:04.774581 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:04.774586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:04.774673 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:04.774770 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:04.776012 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.777132 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.779045 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.779799 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.779938 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.784903 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:04.784911 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:04.784916 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:04.784960 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:04.788825 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.789629 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.789633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.789637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.789798 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.790259 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.790263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.790266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.790667 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:04.790807 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:04.791634 (+ 827us) server_negotiation.cc:299] Negotiation successful
0810 23:38:04.791883 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:38:04.794207 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.773689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.48.252:33209 (local address 127.0.0.1:41024)
0810 23:38:04.773922 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 23:38:04.773941 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:38:04.774047 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:38:04.775033 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:38:04.775038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:38:04.775569 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:38:04.775805 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.775819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.777369 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.777375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.778836 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:38:04.778854 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.780178 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.780184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:38:04.780418 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.781063 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:38:04.781087 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:38:04.784665 (+ 3578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:38:04.788965 (+ 4300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.788970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.788974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.789498 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.789946 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:38:04.789953 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:38:04.789955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:38:04.790117 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:38:04.790867 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:38:04.790871 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:38:04.791000 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 23:38:04.791142 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 23:38:04.791395 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260810 23:38:04.800668 13921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061: Bootstrap starting.
I20260810 23:38:04.804781 13921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061: Neither blocks nor log segments found. Creating new log.
I20260810 23:38:04.806105 13921 log.cc:824] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061: Log is configured to *not* fsync() on all Append() calls
I20260810 23:38:04.815383 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40b0fe0899fe4b61b2574f7ec4191446. Latest consensus state: current_term: 1 leader_uuid: "40b0fe0899fe4b61b2574f7ec4191446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } } }
I20260810 23:38:04.815716 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:38:04.817773 13921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061: No bootstrap required, opened a new log
I20260810 23:38:04.820827 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.772166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.9.48.254:43695)
0810 23:38:04.772375 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:04.772383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:04.793194 (+ 20811us) server_negotiation.cc:407] Connection header received
0810 23:38:04.793716 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:04.793724 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:04.793821 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:04.793898 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:04.798021 (+ 4123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.799246 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.803404 (+ 4158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.804113 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.804224 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.812033 (+ 7809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:04.812043 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:04.812049 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:04.812097 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:04.816357 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.817891 (+ 1534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.817897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.817900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.818083 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.818777 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.818785 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.818790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.819172 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:04.819432 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:04.820130 (+ 698us) server_negotiation.cc:299] Negotiation successful
0810 23:38:04.820399 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:38:04.825722 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40b0fe0899fe4b61b2574f7ec4191446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } } }
I20260810 23:38:04.826011 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:04.829787 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40b0fe0899fe4b61b2574f7ec4191446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } } }
I20260810 23:38:04.830047 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:38:04.833465 13921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } }
I20260810 23:38:04.833875 13921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472b634036a94f888efc4a74f0ebb061, State: Initialized, Role: FOLLOWER
I20260810 23:38:04.834372 13921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } }
I20260810 23:38:04.835390 14090 catalog_manager.cc:1347] Generated new cluster ID: 3475cc8215d7440ca02908c9752d5174
I20260810 23:38:04.835494 14090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:38:04.840870 14109 mvcc.cc:205] Tried to move back new op lower bound from 7317115227188162560 to 7317115227062837248. Current Snapshot: MvccSnapshot[applied={T|T < 7317115227188162560 or (T in {7317115227188162560})}]
I20260810 23:38:04.841331 14125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } } }
I20260810 23:38:04.841758 14125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:38:04.843590 13921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:38:04.843660 13880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40b0fe0899fe4b61b2574f7ec4191446" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "472b634036a94f888efc4a74f0ebb061"
I20260810 23:38:04.844218 13880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:38:04.847870 13880 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40b0fe0899fe4b61b2574f7ec4191446 in term 1.
I20260810 23:38:04.848970 13877 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061 [term 1 FOLLOWER]: Refusing update from remote peer 40b0fe0899fe4b61b2574f7ec4191446: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:38:04.850349 14051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [LEADER]: Connected to new peer: Peer: permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:38:04.852691 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:04.771241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.9.48.252:33209)
0810 23:38:04.771750 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:04.771757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:04.797334 (+ 25577us) server_negotiation.cc:407] Connection header received
0810 23:38:04.797792 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:04.797797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:04.797882 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:04.797948 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:04.802068 (+ 4120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.803216 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.808505 (+ 5289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:04.809598 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:04.809717 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:04.816075 (+ 6358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:38:04.816085 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:38:04.816089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:38:04.816126 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:38:04.819862 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.849448 (+ 29586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.849457 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.849464 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.849685 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:38:04.850300 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:38:04.850309 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:38:04.850314 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:38:04.850854 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:38:04.851211 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:04.851949 (+ 738us) server_negotiation.cc:299] Negotiation successful
0810 23:38:04.852308 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385}
I20260810 23:38:04.832616 13930 master_runner.cc:186] Error getting master registration for 127.9.48.253:37477: OK, instance_id { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" instance_seqno: 1786405084036519 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:38:04.870577 14090 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:38:04.872335 14090 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:38:04.873515 13930 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:38:04.873258 14122 catalog_manager.cc:1259] Loaded cluster ID: 3475cc8215d7440ca02908c9752d5174
I20260810 23:38:04.873811 14122 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170: loading cluster ID for follower catalog manager: success
I20260810 23:38:04.878722 14122 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170: acquiring CA information for follower catalog manager: success
W20260810 23:38:04.880455 14122 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260810 23:38:04.897034 14090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446: Generated new TSK 0
I20260810 23:38:04.897540 14051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40b0fe0899fe4b61b2574f7ec4191446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } } }
I20260810 23:38:04.897840 14051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b0fe0899fe4b61b2574f7ec4191446 [sys.catalog]: This master's current role is: LEADER
I20260810 23:38:04.897745 14125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40b0fe0899fe4b61b2574f7ec4191446. Latest consensus state: current_term: 1 leader_uuid: "40b0fe0899fe4b61b2574f7ec4191446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } } }
I20260810 23:38:04.898058 14125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:38:04.898408 14137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40b0fe0899fe4b61b2574f7ec4191446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40b0fe0899fe4b61b2574f7ec4191446" member_type: VOTER last_known_addr { host: "127.9.48.254" port: 43695 } } peers { permanent_uuid: "472b634036a94f888efc4a74f0ebb061" member_type: VOTER last_known_addr { host: "127.9.48.253" port: 37477 } } peers { permanent_uuid: "e5f08dfe3fe045919387011ea36cf170" member_type: VOTER last_known_addr { host: "127.9.48.252" port: 33209 } } }
I20260810 23:38:04.898806 14137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:38:04.898970 14090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:38:04.959384 14155 catalog_manager.cc:1259] Loaded cluster ID: 3475cc8215d7440ca02908c9752d5174
I20260810 23:38:04.959508 14155 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061: loading cluster ID for follower catalog manager: success
I20260810 23:38:04.963153 14155 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061: acquiring CA information for follower catalog manager: success
I20260810 23:38:04.965582 14155 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 472b634036a94f888efc4a74f0ebb061: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 23:38:05.015830 9411 mini_kdc.cc:348] Time spent kdestroy: real 0.102s user 0.000s sys 0.029s
W20260810 23:38:05.085053 14043 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:05.054906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.9.48.254:43695)
0810 23:38:05.055132 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.055143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.081219 (+ 26076us) server_negotiation.cc:407] Connection header received
0810 23:38:05.083069 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.083076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.083180 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.083426 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:05.084780 (+ 1354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37842: BlockingRecv error: recv got EOF from 127.0.0.1:37842 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 23:38:05.086995 13927 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:05.061349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53160 (local address 127.9.48.253:37477)
0810 23:38:05.061554 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.061560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.085056 (+ 23496us) server_negotiation.cc:407] Connection header received
0810 23:38:05.085353 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.085359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.085471 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.085559 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:05.086759 (+ 1200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53160: BlockingRecv error: recv got EOF from 127.0.0.1:53160 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 23:38:05.088394 14045 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:05.078507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41036 (local address 127.9.48.252:33209)
0810 23:38:05.078703 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.078709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.085998 (+ 7289us) server_negotiation.cc:407] Connection header received
0810 23:38:05.086477 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.086483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.086575 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.086659 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:05.088130 (+ 1471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41036: BlockingRecv error: recv got EOF from 127.0.0.1:41036 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:38:05.161839 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:05.142559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53172 (local address 127.9.48.253:37477)
0810 23:38:05.142767 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.142774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.153431 (+ 10657us) server_negotiation.cc:407] Connection header received
0810 23:38:05.154049 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.154056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.154303 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.154452 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:05.155230 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.156407 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.158524 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.159180 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.159287 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:05.160146 (+ 859us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:05.160929 (+ 783us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:05.161050 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:05.161304 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 23:38:05.161408 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 23:38:05.244139 14045 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:05.226277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.9.48.252:33209)
0810 23:38:05.226478 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.226484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.242400 (+ 15916us) server_negotiation.cc:407] Connection header received
0810 23:38:05.242538 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.242545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.242716 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.242847 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:05.243879 (+ 1032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41044: BlockingRecv error: recv got EOF from 127.0.0.1:41044 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:38:05.292871 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:05.274745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.9.48.253:37477)
0810 23:38:05.275084 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.275094 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.285170 (+ 10076us) server_negotiation.cc:407] Connection header received
0810 23:38:05.285429 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.285436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.285581 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.285698 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:05.286490 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.287837 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.290244 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.291048 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.291169 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:05.291864 (+ 695us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:05.292046 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:05.292120 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:05.292388 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 23:38:05.292486 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
W20260810 23:38:05.347095 14045 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:05.333471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.9.48.252:33209)
0810 23:38:05.333703 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.333711 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.345198 (+ 11487us) server_negotiation.cc:407] Connection header received
0810 23:38:05.345470 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.345477 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.345588 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.345678 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:05.346823 (+ 1145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41056: BlockingRecv error: recv got EOF from 127.0.0.1:41056 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:38:05.397794 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:05.381258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.9.48.253:37477)
0810 23:38:05.381472 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.381480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.390203 (+ 8723us) server_negotiation.cc:407] Connection header received
0810 23:38:05.390753 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.390759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.390871 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.390948 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:05.391695 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.392890 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.394898 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.395605 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.395713 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:05.396706 (+ 993us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:05.396870 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:05.396962 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:05.397346 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 23:38:05.397455 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260810 23:38:05.448729 14045 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:05.439398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58090 (local address 127.9.48.252:33209)
0810 23:38:05.439619 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.439627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.446531 (+ 6904us) server_negotiation.cc:407] Connection header received
0810 23:38:05.447063 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.447070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.447216 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.447306 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:05.448442 (+ 1136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58090: BlockingRecv error: recv got EOF from 127.0.0.1:58090 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 23:38:05.504864 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:05.488306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.9.48.253:37477)
0810 23:38:05.488549 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.488556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.497405 (+ 8849us) server_negotiation.cc:407] Connection header received
0810 23:38:05.497932 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.497938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.498042 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.498122 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:05.498849 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.500126 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.502340 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.503060 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.503161 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:05.503929 (+ 768us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:05.504078 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:05.504145 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:05.504414 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 23:38:05.504552 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260810 23:38:05.547075 14045 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:05.537854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.9.48.252:33209)
0810 23:38:05.538188 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.538196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.545061 (+ 6865us) server_negotiation.cc:407] Connection header received
0810 23:38:05.545590 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.545597 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.545698 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.545778 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:05.546856 (+ 1078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58102: BlockingRecv error: recv got EOF from 127.0.0.1:58102 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
I20260810 23:38:05.565157 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:05.559172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.9.48.253:37477)
0810 23:38:05.559452 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.559459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.559697 (+ 238us) server_negotiation.cc:407] Connection header received
0810 23:38:05.559991 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.560001 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.560079 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.560212 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:05.560759 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.561643 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.563118 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.563632 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.563781 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:05.564249 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:05.564381 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:05.564492 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:05.564700 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 23:38:05.564789 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
W20260810 23:38:05.573725 14045 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:05.571445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58118 (local address 127.9.48.252:33209)
0810 23:38:05.571637 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.571644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.572158 (+ 514us) server_negotiation.cc:407] Connection header received
0810 23:38:05.572343 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.572347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.572420 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.572502 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:05.573469 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58118: BlockingRecv error: recv got EOF from 127.0.0.1:58118 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:38:05.599720 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:05.593599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.9.48.253:37477)
0810 23:38:05.593848 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.593855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.594188 (+ 333us) server_negotiation.cc:407] Connection header received
0810 23:38:05.594433 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.594438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.594514 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.594632 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:05.595079 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.596009 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.598019 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.598462 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.598587 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:05.598945 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:05.599062 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:05.599151 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:05.599283 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 23:38:05.599352 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 23:38:05.608008 14045 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:05.606035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.9.48.252:33209)
0810 23:38:05.606295 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.606301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.606594 (+ 293us) server_negotiation.cc:407] Connection header received
0810 23:38:05.606817 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.606822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.606902 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.607012 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:05.607830 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58122: BlockingRecv error: recv got EOF from 127.0.0.1:58122 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260810 23:38:05.648820 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:05.643608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.9.48.253:37477)
0810 23:38:05.643786 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.643791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.644211 (+ 420us) server_negotiation.cc:407] Connection header received
0810 23:38:05.644468 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.644472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.644574 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.644670 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:05.645082 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.645812 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.647115 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.647602 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.647710 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:05.648057 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:05.648172 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:05.648242 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:05.648415 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 23:38:05.648487 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 23:38:05.656939 14045 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:05.654870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58124 (local address 127.9.48.252:33209)
0810 23:38:05.655086 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.655094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.655458 (+ 364us) server_negotiation.cc:407] Connection header received
0810 23:38:05.655681 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.655685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.655769 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.655885 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:05.656729 (+ 844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58124: BlockingRecv error: recv got EOF from 127.0.0.1:58124 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:38:05.730253 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:05.724920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.9.48.253:37477)
0810 23:38:05.725126 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.725131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.725587 (+ 456us) server_negotiation.cc:407] Connection header received
0810 23:38:05.725718 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.725722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.725803 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.725907 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:05.726324 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.727128 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.728652 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.729085 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.729195 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:05.729615 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:05.729725 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:05.729823 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:05.729945 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 23:38:05.730010 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260810 23:38:05.738240 14045 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:05.736266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.9.48.252:33209)
0810 23:38:05.736505 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.736514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.736849 (+ 335us) server_negotiation.cc:407] Connection header received
0810 23:38:05.737055 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.737062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.737139 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.737271 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:05.738043 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58128: BlockingRecv error: recv got EOF from 127.0.0.1:58128 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 23:38:05.876022 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:05.870345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.9.48.253:37477)
0810 23:38:05.870616 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.870623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.870855 (+ 232us) server_negotiation.cc:407] Connection header received
0810 23:38:05.871118 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.871122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.871194 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.871301 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:05.871817 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.872546 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.874157 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:05.874630 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:05.874739 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:05.875304 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:05.875408 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:05.875483 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:05.875633 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 23:38:05.875702 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260810 23:38:05.883962 14045 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:38:05.882127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58136 (local address 127.9.48.252:33209)
0810 23:38:05.882318 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:05.882322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:05.882647 (+ 325us) server_negotiation.cc:407] Connection header received
0810 23:38:05.882805 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:05.882809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:05.882889 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:05.882972 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:38:05.883706 (+ 734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58136: BlockingRecv error: recv got EOF from 127.0.0.1:58136 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:38:05.891314 14122 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5f08dfe3fe045919387011ea36cf170: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:38:06.150478 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:06.145295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.9.48.253:37477)
0810 23:38:06.145543 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:06.145552 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:06.145822 (+ 270us) server_negotiation.cc:407] Connection header received
0810 23:38:06.146065 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:06.146069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:06.146138 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:06.146240 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:06.146701 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:06.147425 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:06.148786 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:06.149218 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:06.149357 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:06.149758 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:06.149874 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:06.149974 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:06.150132 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 23:38:06.150217 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 23:38:06.162344 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:06.156772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58148 (local address 127.9.48.252:33209)
0810 23:38:06.156993 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:06.156998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:06.157141 (+ 143us) server_negotiation.cc:407] Connection header received
0810 23:38:06.157333 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:06.157337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:06.157503 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:06.157602 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:06.157977 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:06.158692 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:06.160079 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:06.160515 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:06.160665 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:06.161141 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:06.161718 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:06.161831 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:06.161935 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 23:38:06.162008 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:38:06.180065 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:06.168119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.9.48.252:33209)
0810 23:38:06.168548 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:06.168556 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:06.173782 (+ 5226us) server_negotiation.cc:407] Connection header received
0810 23:38:06.174041 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:06.174049 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:06.174137 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:06.174234 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:06.175070 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:06.175967 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:06.178010 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:06.178577 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:06.178706 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:06.179261 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:06.179397 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:06.179490 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:06.179655 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 23:38:06.179747 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260810 23:38:06.180253 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:06.168111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59578 (local address 127.9.48.253:37477)
0810 23:38:06.168339 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:06.168348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:06.173485 (+ 5137us) server_negotiation.cc:407] Connection header received
0810 23:38:06.173851 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:06.173858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:06.173987 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:06.174057 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:06.174975 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:06.176021 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:06.178221 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:06.178843 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:06.178989 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:06.179432 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:06.179568 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:06.179655 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:06.179858 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 23:38:06.179937 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:38:06.180406 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:38:06.168491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.9.48.254:43695)
0810 23:38:06.173027 (+ 4536us) server_negotiation.cc:206] Beginning negotiation
0810 23:38:06.173036 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:38:06.173062 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:38:06.173206 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:38:06.173214 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:38:06.173476 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:38:06.173690 (+ 214us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:38:06.174282 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:06.175431 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:06.177650 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:38:06.178300 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:38:06.178422 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:38:06.179027 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:38:06.179552 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:38:06.179972 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0810 23:38:06.180014 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 23:38:06.180079 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4376,"thread_start_us":4171,"threads_started":1}
I20260810 23:38:06.185703 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 13692
I20260810 23:38:06.186897 13787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:06.297595 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 13692
I20260810 23:38:06.325368 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 13804
I20260810 23:38:06.326598 13911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:06.433698 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 13804
I20260810 23:38:06.463542 9411 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskahRJgu/build/asan/bin/kudu pid 13930
I20260810 23:38:06.464748 14030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 23:38:06.572158 9411 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskahRJgu/build/asan/bin/kudu with pid 13930
2026-08-10T23:38:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5247 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10183 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (387964 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST